Dynamic Analysis of an Amensalism Model Driven by Multiple Factors: The Interwoven Impacts of Refuge, the Fear Effect, and the Allee Effect

Abstract

This paper investigates a model of amensalism, in which the first species is influenced by the combined effects of refuge and fear, while the second species exhibits an additive Allee effect. The paper analyzes the existence and stability of the equilibria of the system and derives the conditions for various bifurcations. In the global structure analysis, the stability at infinity is examined, and the phenomena of global stability and bistability in the system are analyzed. Additionally, a sensitivity analysis is employed to evaluate the impact of system parameters on populations. The study reveals that refuge has a significant positive effect on the first population, and refuge’s effect becomes more pronounced as the fear level increases. Under the strong Allee effect, when the initial density of the second species is low, the second species may eventually become extinct; when the initial density is high, if the refuge parameter is below a certain threshold, increasing the refuge parameter slows down the extinction of the first species, whereas, when the refuge parameter exceeds this threshold, the two species can coexist. Under the weak Allee effect, when the refuge parameter surpasses a certain threshold, the two species can achieve long-term, stable coexistence, and the threshold for the weak Allee effect is higher than that for the strong Allee effect.

1. Introduction

In nature, amensalism is an interspecies interaction through which one species exerts a negative impact on another without being affected itself. This type of interaction is common in plant, microbial, and animal communities [1]. For example, in the alpine meadows of Tibet, Xi et al. [2] found a strong amensalism effect between two major herbivorous insects, grasshoppers and caterpillars. Although the presence of grasshoppers did not directly result in predation on caterpillars, the grasshoppers’ jumping behavior triggered a “tonic immobility” response in the caterpillars, reducing their foraging efficiency and reproductive capacity. This indirect effect ultimately negatively impacted the caterpillar population density. On the African savanna, large herbivores like elephants, through trampling and compacting the soil, unintentionally harm or even kill small soil-dwelling arthropods, but this has no effect on the elephants themselves [3]. In the Australian forest ecosystem, although ants do not directly feed on certain plants, their nest construction and resource competition indirectly affect plant growth. Ant activities damage plant roots, reducing their growth potential and negatively impacting plant populations, but this behavior has no impact on the ants themselves [4]. In the tropical ecosystems of East Africa, wasps completely avoid using the cavities previously occupied by bees, while bees occupy any cavity, regardless of whether it was previously used by wasps. This is another clear example of amensalism [5].

In the last ten years, remarkable progress has been achieved in the research on amensalism relationships and population dynamics models. The pioneering model of interspecific amensalism was originally introduced by Sun [6], which describes the dynamics as follows

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}& =r_{1}x\left({\displaystyle \frac{k_1-x-cy}{k_1}}\right),\hfill \\ \hfill {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}& =r_{2}y\left({\displaystyle \frac{k_2-y}{k_2}}\right).\hfill \end{array}\right.$$

(1)

Here, $x\left(t\right)$ and $y\left(t\right)$ stand for the population densities of the first and second species at time t, respectively. The parameters $r_1$ and $r_2$ represent the intrinsic growth rates of the two species, and $k_1$ and $k_2$ are used to denote their respective environmental carrying capacities. Moreover, the coefficient c reflects the influence of one individual of species y on species x.

In the following years, many scholars proposed various models on amensalism and studied their dynamical behaviors [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22]. For example, scholars have extended amensalism models by incorporating Michaelis–Menten-type harvesting [7], the fear effect [10], Beddington–DeAngelis functional responses [9,10,12], non-selective harvesting [13], or nonlinear birth rates [11].

In recent ecological modeling studies, the impact of fear effects on interspecies relationships and population dynamics has become a focal research topic. The fear effect specifically refers to behavioral patterns and physiological changes induced in prey species upon detecting predator presence (even in the absence of actual predation), which significantly affect their survival capacity and reproductive success [23,24]. Notably, this phenomenon has been validated not only in typical predator–prey relationships but also in amensalistic interactions [10,25,26,27,28,29]. Researchers led by Xi [2] identified an amensalistic relationship between grassland caterpillars and grasshoppers. Through carefully designed controlled experiments, they established four distinct experimental groups: plant ecosystems containing only grasshoppers, those containing only caterpillars, composite ecosystems containing both grasshoppers and caterpillars, and blank control groups completely devoid of both species. A systematic comparison of oviposition success rates and larval survival rates across groups revealed that in grasshopper–caterpillar coexistence groups, caterpillar reproductive indices were significantly lower than in single-species groups. This crucial finding demonstrates that grasshopper presence can substantially inhibit caterpillar reproductive efficiency through fear effects, ultimately leading to sharp population declines. These results provide solid experimental foundation for incorporating fear effect theory into amensalism research. Building upon these findings, Zhu et al. [10] developed an amensalism model incorporating fear effects:

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}& =x\left({\displaystyle \frac{a_1}{1+k_{1}y}}-d_1-b_{1}x-{\displaystyle \frac{cxy}{1+mx+ny}}\right),\hfill \\ \hfill {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}& =y\left(a_2-b_{2}y\right).\hfill \end{array}\right.$$

(2)

Their study demonstrated that, under specific parameter conditions, when fear intensity remains below a critical threshold, increased fear effects only reduce the population density of the first species without altering its survival status; however, when fear effects exceed this threshold, any further intensification accelerates the extinction process of the first species.

Refuges typically refer to areas where predators cannot enter or have significantly reduced predation efficiency, providing a relatively safe habitat for prey. In many ecosystems, the presence of refuges plays a crucial role in population stability and dynamic behavior. However, most studies on refuges have focused on predator–prey systems [28,29]. It was not until recently that researchers began to explore the impact of refuges on amensalism models [7,14,30,31,32,33]. Introducing the concept of refuges into amensalism models can help describe how weaker species utilize refuge mechanisms to withstand amensal pressures, thereby altering species interactions and dynamic behavior. Xie et al. [14] took into account an amensalism model in which the first species has a refuge. The model is presented in the following form:

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}& =a_{1}x-b_1{x}^2-c_1(1-k)xy,\hfill \\ \hfill {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}& =a_{2}y-b_2{y}^2.\hfill \end{array}\right.$$

(3)

In their analysis, the authors of the study considered a refuge parameter, k, representing the proportion of the habitat inaccessible to the second species, constrained by $0<k<1$. The primary emphasis was placed on assessing the global stability of equilibria. It was determined that ensuring the local stability of the boundary equilibrium, $E_2$, is adequate to establish its global stability. Furthermore, the existence of a positive equilibrium was found to inherently confer global stability. This study offers a significant theoretical foundation for comprehending the role of refuges in amensalism systems. However, this study only considered the single refuge effect and overlooked other factors that are equally important in natural ecosystems.

Afterward, Wu [18] put forward an amensalism model incorporating a Holling type II functional response and a refuge, and then they explored the local and global stability of all possible equilibria within the system. Subsequently, Liu et al. [7] introduced an amensalism model that incorporates Michaelis–Menten-type predation for the first species, alongside a refuge mechanism. The study meticulously analyzed the system’s equilibria, employing the Sotomayor theorem to reveal the occurrence of two saddle-node and two transcritical bifurcations under appropriate parameter regimes. Chong et al. [31] proposed an amensalism model with a saturation effect in the refuge, where the refuge was dependent on the density of the second species. For the autonomous case, they pinpointed specific threshold conditions essential for predicting the extinction or stable persistence of the first species. For the non-autonomous case, they established sufficient conditions guaranteeing system persistence, global asymptotic stability, and species extinction. Huang and Chen [33] investigated an amensalism model where the first species exhibits an Allee effect in its intrinsic growth rate and possesses a refuge. The research indicated that alterations in system parameters influence both the quantity and stability of equilibria, culminating in saddle-node bifurcations. Furthermore, the Allee effect was shown to negatively impact the growth of the first species, with an increasing Allee effect intensity accelerating the extinction rate of the first species.

Previous studies have primarily focused on the impact of a single refuge on the first population in amensalism models [7,14,31,32,33]. However, in the experiment conducted by Osakabe et al. [34], it was found that T. urticae exerts an amensal effect on P. ulmi. The invasion of T. urticae inhibits the population growth of P. ulmi. Interestingly, the highly asymmetric distribution of T. urticae on the leaf surface enables adult P. ulmi to use the upper leaf surface as a refuge, facilitating the coexistence of both species on the same leaf. The experiment demonstrated that in the presence of T. urticae, the population growth of P. ulmi was significantly inhibited, while in the control group without T. urticae, the P. ulmi population continued to grow unimpeded. This behavioral response can be interpreted as a “fear” effect of P. ulmi in response to T. urticae. Specifically, P. ulmi individuals residing on the upper leaf surface (the refuge) appear unaffected by this fear effect, whereas those inhabiting the lower leaf surface (outside the refuge) are subject to its influence. These findings motivate us to investigate the combined effects of both refuge and fear effect on the first population in amensalism models.

The Allee effect, a phenomenon in ecological and population dynamics, refers to the observation that a population may experience reduced growth or even extinction when its density falls below a certain threshold [35,36]. This effect is particularly significant in the context of species interactions, where the presence of a minimal population size is critical for survival and reproduction. In amensalism systems, the Allee effect can significantly alter the dynamics, influencing the competition and survival rates of species. Scholars have lately begun studying this impact [9,11,12,15,17,19,20]. Guan and Chen [9] considered an amensalism system in which the second species exhibits the Allee effect. Their results indicate that, compared to a system without the Allee effect, the system with the Allee effect requires a longer time to reach a stable steady-state solution. Subsequently, Guo et al. [37] explored an amensalism model in which the first species exhibits a pronounced strong Allee effect. Their findings suggest that low population densities of the first species can lead to extinction due to the Allee effect. Additionally, an increase in the Allee effect parameter accelerates the system’s convergence to a stable equilibrium. Dennis [38] first proposed a model incorporating the additive Allee effect, formulated as follows:

$${\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}=rx\left(\left(1-{\displaystyle \frac{x}{k}}\right)-{\displaystyle \frac{n}{x+a}}\right).$$

The additive Allee effect demonstrates threshold behavior: for parameter values $0<n<a$, the effect is considered weak, while values where $n>a$ signify a strong Allee effect. In the experiments conducted by Osakabe et al. [34], the population density of T. urticae declined in later stages due to leaf senescence, leading to reduced population numbers. At low population densities, mating difficulties among individuals resulted in restricted population growth, which prompted us to consider the impact of the additive Allee effect on the second species.

Inspired by the groundbreaking studies of Osakabe et al. [34] and Xie et al. [14], we are naturally led to a fundamental research question: how to construct a mathematical model to describe the scenario where the fear effect is influenced by refuge while the invading population is subject to the Allee effect, and how will the system’s dynamic behavior change under these conditions? To address this, we develop an improved model based on system (3) that incorporates both refuge-dependent fear effects and an additive Allee effect in the second species. The paper is organized as follows: In Section 2, we formulate a novel mathematical model for amensalism. Section 3 not only establishes sufficient conditions for the existence and stability of all possible equilibria but also provides a rigorous analysis of the key conditions governing saddle-node bifurcation, pitchfork bifurcation, and transcritical bifurcation. Furthermore, we investigate the stability of the system under infinity, derive global stability and bistability conditions for the equilibria, and perform a sensitivity analysis of the model solutions. In Section 4, numerical simulations validate the dynamical behaviors under both strong and weak Allee effects, revealing distinct dynamic characteristics. Finally, Section 5 presents the conclusions and discusses potential implications.

2. Construction of the Mathematical Model

2.1. Model Assumptions

1.

Refuge and fear effect for the first species.

(a) Refuge structure: A proportion, m, of the population resides within the refuge, where they are entirely unaffected by the fear effect and the second species; the remaining proportion, $(1-m)$, of individuals are exposed outside the refuge;

(b) Fear effect formulation: Individuals outside the refuge perceive a fear level k from the second species, which reduces their birth rate to ${\displaystyle \frac{r_1(1-m)}{1+ky}}$. This indicates that the fear effect suppresses reproduction in an inversely proportional manner but does not directly increase mortality. In contrast, individuals inside the refuge remain unaffected by the fear effect, maintaining a birth rate of $r_{1}m$;

(c) Interspecific inhibition: the direct impact of the second species on individuals outside the refuge is described by the linear term $-c(1-m)xy$, reflecting resource competition or direct interference.

2.

The Allee effect for the second species.

Additive Allee effect: In the second population, the additive Allee effect term $-{\displaystyle \frac{n}{y+a}}$ causes significant population decline under low-density conditions ($y\to 0$). Then, the second population density dynamics is governed by the equation ${\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}=y\left(e-fy-{\displaystyle \frac{n}{y+a}}\right)$; when the condition $n>ae$ is met, the system exhibits a strong Allee effect, meaning that there exists a critical threshold density below which the population cannot sustain growth. Conversely, when $0<n<ae$, the system shows a weak Allee effect through which population growth is only suppressed but not completely terminated.

3.

Amensalism relationship.

Unidirectional inhibition: the second species inhibits the first species through the fear effect and direct interaction, but the first species has no influence on the dynamics of the second species (i.e., the first species variable x does not appear in the second equation).

4.

Parameter constraints.

Non-negativity and biological relevance: all parameters, $r_1,r_2,b,c,e,f,n,a,k>0$, $r_1>r_2$, and $m\in [0,1]$, ensure the biological plausibility of the model solutions.

5.

Simplifying assumptions.

(a) No migration or age structure: the populations are assumed to be closed, with no immigration or emigration, and the age or stage structure is ignored;

(b) Time delays and environmental fluctuations: time delays in reproduction or interactions are not considered, and environmental parameters (e.g., resources, refuge proportion) are assumed to be constant.

2.2. Model Formulation

This study is the first to consider the combined effects of the refuge proportion m and fear level k on the birth rate.

Define the birth rate function as

$$r=r_{1}m+{\displaystyle \frac{r_1(1-m)}{1+ky}}.$$

As m increases, more individuals are sheltered in the refuge, unaffected by the fear effect, leading to an increase in the birth rate; as k increases, the fear effect intensifies, reducing the birth rate of individuals outside the refuge.

By fixing $r_1=1$ and $y=1$, Figure 1 illustrates the effects of m and k on r: increasing the refuge proportion mitigates the negative impact of the fear effect on the birth rate. Moreover, the higher the fear level, the more pronounced the enhancing effect of the refuge proportion on the birth rate.

From an ecological perspective, refuges provide a safe habitat for populations, reducing the number of individuals exposed to amensalistic threats. When the refuge proportion m increases, more individuals are sheltered within the refuge, shielding them from the direct impact of the fear effect and thereby enhancing the birth rate. Additionally, refuges play a more critical role in high-fear environments by providing individuals with a safe space, significantly mitigating the inhibitory effects of fear on reproductive behavior. When fear levels are low, the role of shelters may be relatively limited; however, as fear levels increase, the importance of refuges becomes more pronounced, enabling them to more effectively alleviate stress and enhance reproductive success rates.

From the above analysis, we propose an amensalism model where the first species is affected by the fear effect and has a refuge, while the second species exhibits an additive Allee effect. The model is expressed as follows:

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{d}x}{\mathrm{d}\tau }}& =x\left(r_{1}m+{\displaystyle \frac{r_1(1-m)}{1+ky}}-r_2-bx\right)-c(1-m)xy,\hfill \\ \hfill {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}\tau }}& =y\left(e-fy-{\displaystyle \frac{n}{y+a}}\right).\hfill \end{array}\right.$$

(4)

Table 1. Parameters, symbols, dimensions, and ecological meanings.

Parameter	Symbol	Dimension	Ecological Meaning
First species density	x	$\left[N_1\right]$	Population density of the first species at time $\tau$
Second species density	y	$\left[N_2\right]$	Population density of the second species at time $\tau$
Refuge proportion	m	Dimensionless	Proportion of protected habitat reducing amensalistic interactions, $m\in [0,1]$
Fear level	k	$[1/N_2]$	Fear level of the first species towards the second species
Birth rate	$r_1$	$[1/T]$	Birth and rate of the first species
Mortality rate	$r_2$	$[1/T]$	Mortality rate of the first species
Intraspecific competition	b	$[1/\left(N_{1}T\right)]$	Density-dependent resource limitation within the first species
Intraspecific competition	f	$[1/\left(N_{2}T\right)]$	Density-dependent resource limitation within the second species
Impact parameter	c	$[1/\left(N_{2}T\right)]$	Strength of amensalistic effect per encounter
Intrinsic growth rate	e	$[1/T]$	Maximum growth rate of the second species under ideal conditions
Allee threshold	a	$\left[N_2\right]$	Additive Allee effect parameter governing the strength and shape of the effect
Allee intensity	n	$[N_2/T]$	Additive Allee effect parameter

summarizes the symbols and ecological meanings of the parameters in the system (4). The detailed descriptions in the table serve as an important reference for subsequent analysis and discussion.

In order to reduce the number of parameters in system (4), the system employs a set of dimensionless variables, as outlined below.

$${\displaystyle \frac{b}{r_1}}x=\overline{x},{\displaystyle \frac{y}{a}}=\overline{y},r_1\tau =t,ak=\overline{k},m=\overline{m},{\displaystyle \frac{r_2}{r_1}}=\overline{d},{\displaystyle \frac{ac}{r_1}}=\overline{c},{\displaystyle \frac{e}{r_1}}=\overline{e},{\displaystyle \frac{af}{e}}=\overline{f},{\displaystyle \frac{n}{ae}}=\overline{n}.$$

Upon removing the bars, the system described in Equation (4) can be reformulated as follows:

$$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t}}& =x\left(m+{\displaystyle \frac{1-m}{1+ky}}-d-x-c(1-m)y\right),\hfill \\ \hfill {\displaystyle \frac{\mathrm{d}y}{\mathrm{d}t}}& =ey\left(1-fy-{\displaystyle \frac{n}{1+y}}\right).\hfill \end{array}\right.$$

(5)

It should be noted that after non-dimensionalization, the parameter m still remains within the range $[0,1]$, while the condition $r_1>r_2$ in system (4) translates to $d<1$ in system (5). When $n>1$, the system exhibits a strong Allee effect, whereas, when $n<1$, it demonstrates a weak Allee effect.

This study innovatively incorporates both the refuge and fear effect into a two-species amensalism model. Previous studies mostly considered only one of two factors separately. This model more comprehensively reflects the complex interactions among species in natural ecosystems, providing a more realistic model framework for the study of amensalism systems. The research has discovered the influence rules of the refuge parameter on species coexistence and extinction under different intensities of the Allee effect. Under a weak Allee effect, an appropriate refuge parameter can promote the long-term stable coexistence of the two species. Under a strong Allee effect, the initial density and the refuge parameter jointly determine the fate of the species. These results reveal new ecological phenomena and provide a new theoretical basis for ecological protection and management.

3. Global Dynamics of System (5)

3.1. Existence of Equilibria

When discussing the existence of equilibria for system (5), by substituting ${\displaystyle \frac{dx}{dt}}={\displaystyle \frac{dy}{dt}}=0$ into system (5), the following equations are derived:

$$\left\{\begin{array}{c}{\displaystyle x\left(m+\frac{1-m}{1+ky}-d-x-c(1-m)y\right)=0,}\hfill \\ {\displaystyle ey\left(1-fy-\frac{n}{y+1}\right)=0.}\hfill \end{array}\right.$$

(6)

From (6), it is evident that system (5) has two boundary equilibria, namely $E_0(0,0)$ and $E_d(1-d,0)$.

To find the other boundary, $E_i(0,y_i)$, and the positive equilibria, $E_i^{*}(x_i^{*},y_i)$, of system (5), we focus on the positive solutions of the subsequent equations:

$$\left\{\begin{array}{c}{\displaystyle x=m+\frac{1-m}{1+ky}-d-c(1-m)y,}\hfill \\ {\displaystyle f{y}^2+(f-1)y+n-1=0.}\hfill \end{array}\right.$$

(7)

We define the second equation in (7) as

$$F\left(y\right)=f{y}^2+(f-1)y+n-1.$$

(8)

The discriminant of the quadratic function $F\left(y\right)$ is calculated as follows:

$$\begin{array}{cc}\hfill {\Delta }_1\left(f\right)& =f^2+(2-4n)f+1.\hfill \end{array}$$

And we obtain the discriminant for the equation ${\Delta }_1=0$ as ${\Delta }_2=16n^2-16n$.

We first present a theorem regarding the root conditions of $F\left(y\right)$.

Theorem 1.

Regarding the function $F\left(y\right)$,

(1) 

When $n<1$, $F\left(y\right)$ has one positive root $y_1$;

(2) 

When $n=1$ and $f<1$, $F\left(y\right)$ has one positive root $y_1$;

(3) 

When $n>1$

(i)   

and $0<f<f^{-}$, $F\left(y\right)$ has two positive roots, $y_1$ and $y_2$;

(ii)   

and $f=f^{-}$, $F\left(y\right)$ has one positive, root $y_3$.

where $f^{-}={\left(\sqrt{n}-\sqrt{n-1}\right)}^2$, $y_1={\displaystyle \frac{1-f+\sqrt{{\Delta }_1}}{2f}}$, $y_2={\displaystyle \frac{1-f-\sqrt{{\Delta }_1}}{2f}}$ and $y_3={\displaystyle \frac{1-f}{2f}}$.

Proof. 

If ${\Delta }_1>0$, then $F\left(y\right)$ has two roots, $y_1={\displaystyle \frac{1-f+\sqrt{{\Delta }_1}}{2f}}$, $y_2={\displaystyle \frac{1-f-\sqrt{{\Delta }_1}}{2f}}$, and if ${\Delta }_1=0$, $F\left(y\right)$ has a unique root, $y_3={\displaystyle \frac{1-f}{2f}}$. Thus, when $n<1$, then $y_1>0,y_2<0$; when $n=1$ and $f<1$, then $y_1>0,y_2=0$; when $n=1,f\ge 1$, then $y_1\le 0,y_2=0$; when $n>1$, we need to analyze the positivity and negativity of discriminants ${\Delta }_1$. We have ${\Delta }_2>0$, and then the equation ${\Delta }_1=0$ yields two distinct positive roots:

$$f^{-}={\left(\sqrt{n}-\sqrt{n-1}\right)}^2,f^{+}={\left(\sqrt{n}+\sqrt{n-1}\right)}^2.$$

In this case, when $0<f<f^{-}$ or $f>f^{+}$, ${\Delta }_1>0$, when $f^{-}<f<f^{+}$, ${\Delta }_1<0$, and when $f=f^{-}$ or $f=f^{+}$, ${\Delta }_1=0$. Furthermore, we can easily obtain $f^{-}<1<f^{+}$. Therefore, based on the above analysis, we can derive the root distribution when $n>1$. When $0<f<f^{-}({\Delta }_1>0)$, then $y_1>0,y_2>0$. When $f=f^{-}({\Delta }_1=0)$, then $y_3>0$. When $f^{-}<f<f^{+}({\Delta }_1<0)$, then $F\left(y\right)$ has no roots. When $f\ge f^{+}({\Delta }_1\ge 0)$, then $y_1<0,y_2<0$.

The proof of Theorem 1 is completed.  □

To better visualize the root conditions of $F\left(y\right)$, we can decompose $F\left(y\right)$ into two components: $1-fy$ and ${\displaystyle \frac{n}{1+y}}$. The roots of $F\left(y\right)$ then correspond to the intersection points of these two functions, as illustrated in Figure 2.

Theorem 2.

System (5) consistently has two boundary equilibria, $E_0(0,0)$ and $E_d(1-d,0)$. The subsequent results pertain to the other boundary equilibria and positive equilibria of system (5).

(1) 

For $0<n<1$ (weak Allee effect);

(a)   

When $m>g\left(y_1\right)$, then system (5) has one boundary equilibrium, $E_1(0,y_1)$, and one positive equilibrium, $E_1^{*}(x_1^{*},y_1);$

(b)   

When $m\le g\left(y_1\right)$, then system (5) has one boundary equilibrium, $E_1(0,y_1)$.

(2) 

For $n=1$,

(a)   

when $f<1$

(i)   

and $m>g\left(y_1\right)$, then system (5) has one boundary equilibrium, $E_1(0,y_1)$, and one positive equilibrium, $E_1^{*}(x_1^{*},y_1);$

(ii)   

and $m\le g\left(y_1\right)$, then system (5) has one boundary equilibrium, $E_1(0,y_1);$

(b)   

when $f\ge 1$, then system (5) lacks the other equilibria.

(3) 

For $n>1$ (strong Allee effect),

(a)   

when $0<f<f^{-}$

(i)   

and $m>g\left(y_1\right)$, then system (5) has two boundary equilibria, $E_1(0,y_1),E_2(0,y_2)$, and two positive equilibria, $E_1^{*}(x_1^{*},y_1)$, $E_2^{*}(x_2^{*},y_2)$;

(ii)   

and $g\left(y_2\right)<m\le g\left(y_1\right)$, then system (5) has two boundary equilibria, $E_1(0,y_1)$ and $E_2(0,y_2)$, and one positive equilibrium, $E_2^{*}(x_2^{*},y_2)$;

(iii)   

and $m\le g\left(y_2\right)$, then system (5) has two boundary equilibria, $E_1(0,y_1)$ and $E_2(0,y_2)$;

(b)   

when $f=f^{-}$,

(i)   

and $m>g\left(y_3\right)$, then system (5) has one boundary equilibrium, $E_3(0,y_3)$, and one positive equilibrium, $E_3^{*}(x_3^{*},y_3)$;

(ii)   

and $m\le g\left(y_3\right)$, then system (5) has one boundary equilibrium, $E_3(0,y_3)$;

(c)   

When $f>f^{-}$, then system (5) lacks the other equilibria.

Proof. 

According to (7), we must ensure that $x_i^{*}>0$, and then, $m>{\displaystyle \frac{ck{y_i}^2+dk{y}_i+c{y}_i+d-1}{y_i\left(ck{y}_i+c+k\right)}}\triangleq g\left(y_i\right)$. Since ${\displaystyle \frac{\partial g\left(y_i\right)}{\partial y_i}}={\displaystyle \frac{\left(1-d\right)\left({\left(ky+1\right)}^{2}c+k\right)}{y^2{\left(cky+c+k\right)}^2}}>0$ and $y_1>y_2$, we can simply confirm that $g\left(y_1\right)>g\left(y_2\right)$.

Hence, it can be concluded that

($H_1$) If $m>g\left(y_1\right)$, $x_1^{*}>0,x_2^{*}>0.$

($H_2$) If $g\left(y_2\right)<m\le g\left(y_1\right)$, $x_1^{*}\le 0,x_2^{*}>0.$

($H_3$) If $m\le g\left(y_2\right)$, $x_1^{*}<0,x_2^{*}\le 0.$

For Theorem 1 and the conditions of ($H_1$), ($H_2$), and ($H_3$), it is straightforward to derive the following conclusions:

(1) 

When $0<n<1$, then $x_1^{*}>0$ if $m>g\left(y_1\right)$, and $x_1^{*}\le 0$ if $m\le g\left(y_1\right)$.

(2) 

When $n=1$ and $f<1$, $x_1^{*}>0$ if $m>g\left(y_1\right)$, and $x_1^{*}\le 0$ if $m\le g\left(y_1\right)$.

(3) 

When $n>1$

(a)   

and $0<f<f^{-}$, $x_1^{*}>0,x_2^{*}>0$ if $m>g\left(y_1\right)$, $x_1^{*}\le 0,x_2^{*}>0$ if $g\left(y_2\right)<m\le g\left(y_1\right)$, and $x_1^{*}<0,x_2\le 0$ if $m\le g\left(y_2\right)$;

(b)   

and $f=f^{-}$, $x_3^{*}>0$ if $m>g\left(y_3\right)$, and $x_3^{*}\le 0$ if $m\le g\left(y_3\right)$.

The proof of Theorem 2 is completed.  □

3.2. Stability of Equilibria

Theorem 3.

For the boundary equilibria $E_0(0,0)$ and $E_d(1-d,0)$ of system (5),

(1) 

When $0<n<1$, $E_0$ is an unstable node and $E_d$ is a saddle.

(2) 

When $n>1$, $E_0$ is a saddle and $E_d$ is a stable node.

(3) 

When $n=1$,

(i)   

$f\ne 1$, $E_0$ is a repelling saddle-node, and $E_d$ is an attracting saddle-node;

(ii)   

$f=1$, $E_0$ is a saddle of codimension 2, and $E_d$ is an unstable node.

Proof. 

The Jacobian matrix of system (5) is

$$J\left(E\right)=\left(\begin{array}{cc}{\mathsf{\Lambda }}_1& {\mathsf{\Lambda }}_2\\ 0& {\mathsf{\Lambda }}_3\end{array}\right),$$

(9)

where

$$\begin{array}{cc}\hfill {\mathsf{\Lambda }}_1& =m+{\displaystyle \frac{1-m}{ky+1}}-d-2x-c\left(1-m\right)y,\hfill \\ \hfill {\mathsf{\Lambda }}_2& =x\left(-{\displaystyle \frac{\left(1-m\right)k}{{\left(ky+1\right)}^2}}-c(1-m)\right),\hfill \\ \hfill {\mathsf{\Lambda }}_3& =\left(1-fy-{\displaystyle \frac{n}{1+y}}\right)e+ey\left(-f+{\displaystyle \frac{n}{{\left(1+y\right)}^2}}\right).\hfill \end{array}$$

The Jacobian matrix at $E_0(0,0)$ is

$$J\left(E_0\right)=\left(\begin{array}{cc}1-d& 0\\ 0& (1-n)e\end{array}\right),$$

(10)

the two eigenvalues of $J\left(E_0\right)$ are expressed as follows: ${\lambda }_1=1-d>0$ and ${\lambda }_2=(1-n)e$. Clearly, if $0<n<1$, then ${\lambda }_2>0$, indicating that $E_0$ is an unstable node. Conversely, if $n>1$, then ${\lambda }_2<0$, making $E_0$ a saddle. When $n=1$, then ${\lambda }_2=0$, making $E_0$ a degenerate equilibrium. Expanding system (5) around the origin up to the third order and making the followigng time transformation: $d\tau =(1-d)dt.$ The resulting system (5) is as follows:

$$\left\{\begin{array}{c}{\displaystyle \frac{dx}{d\tau }=x-\frac{1}{1-d}{x}^2+\frac{km-k+cm-c}{1-d}xy+O\left({\left|x,y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dy}{d\tau }=\frac{e(1-f)}{1-d}{y}^2-\frac{e}{1-d}{y}^3+O\left({\left|x,y\right|}^3\right).}\hfill \end{array}\right.$$

(11)

Notice that, if $f\ne 1$, then ${\displaystyle \frac{e(1-f)}{1-d}}\ne 0$. According to Theorem 7.1 in Zhang et al. [39], $E_0$ is a repelling saddle-node, which consists of two hyperbolic sectors and one parabolic sector. Specifically, when $0<f<1$, the parabolic sector lies above the x-axis, while the two hyperbolic sectors are situated in the third and fourth quadrants, respectively. When $f>1$, the parabolic sector lies below the x-axis with the two hyperbolic sectors occupying the first and second quadrants, respectively. If $f=1$, then ${\displaystyle \frac{e(1-f)}{1-d}}=0$, it shows that $E_0$ is a saddle of codimension 2.

The Jacobian matrix at $E_d(1-d,0)$ is

$$J\left(E_d\right)=\left(\begin{array}{cc}d-1& -(d-1)(m-1)(c+k)\\ 0& (1-n)e\end{array}\right),$$

(12)

the two eigenvalues of $J\left(E_d\right)$ are expressed as follows: ${\lambda }_1=d-1<0$ and ${\lambda }_2=(1-n)e$. Clearly, if $0<n<1$, then ${\lambda }_2>0$, indicating that $E_d$ is a saddle. Conversely, if $n>1$, then ${\lambda }_2<0$, making $E_d$ a stable node. When $n=1$, then ${\lambda }_2=0$, making $E_d$ a degenerate equilibrium. Expanding system (5) around the origin up to the third order and make the follow transformation:

$$\begin{array}{cc}& x=X+(1-d),y=Y,\hfill \\ & X=x+(m-1)(k+c)y,Y=y,\tau =(d-1)t.\hfill \end{array}.$$

The resulting system (5) is as follows:

$$\left\{\begin{array}{c}{\displaystyle \frac{dx}{d\tau }=x-\frac{1}{d-1}{x}^2+\frac{(1-m)(k+c)}{d-1}xy+O\left({\left|x,y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dy}{d\tau }=\frac{e(1-f)}{d-1}{y}^2-\frac{e}{d-1}{y}^3+O\left({\left|x,y\right|}^3\right).}\hfill \end{array}\right.$$

(13)

Notice that, if $f\ne 1$, then ${\displaystyle \frac{e(1-f)}{d-1}}\ne 0$. According to Theorem 7.1 in Zhang et al. [39] and the applied time transformation, $E_d$ is an attracting saddle-node with two hyperbolic sectors and one parabolic sector. Specifically, when $0<f<1$, the parabolic sector lies below the x-axis, while the two hyperbolic sectors are situated in the first and second quadrants, respectively. When $f>1$, the parabolic sector lies above the x-axis with the two hyperbolic sectors occupying the third and fourth quadrants, respectively. If $f=1$, then ${\displaystyle \frac{e(1-f)}{d-1}}=0$. Obviously, $E_d$ is an unstable node.

The proof of Theorem 3 is completed.  □

Theorem 4.

The stability of the other boundary equilibria for system (5) is shown below:

(1) 

For $E_1(0,y_1)$, when $E_1$ exists (i.e., $0<n<1$ or $n=1$ and $f<1$ or $n>1$ and $0<f<f^{-}$),

(i) 

and $m>g\left(y_1\right)$, $E_1$ is a saddle;

(ii) 

and $m<g\left(y_1\right)$, $E_1$ is a stable node;

(iii) 

and $m=g\left(y_1\right)$, $E_1$ is an attracting saddle-node.

(2) 

For $E_2(0,y_2)$, when $E_2$ exists (i.e., $n>1$ and $0<f<f^{-}$),

(i) 

and $m>g\left(y_2\right)$, $E_2$ is an unstable node;

(ii) 

and $m<g\left(y_2\right)$, $E_2$ is a saddle;

(iii) 

and $m=g\left(y_2\right)$, $E_2$ is a repelling saddle-node.

(3) 

For $E_3(0,y_3)$, when $E_3$ exists (i.e., $n>1$ and $f=f^{-}$),

(i) 

and $m>g\left(y_3\right)$, $E_3$ is a repelling saddle-node;

(ii) 

and $m<g\left(y_3\right)$, $E_3$ is an attracting saddle-node;

(iii) 

and $m=g\left(y_3\right)$, $E_3$ is a saddle.

Proof. 

The Jacobian matrix at $E_i(0,y_i)$ is

$$J\left(E_i\right)=\left(\begin{array}{cc}{\displaystyle m+\frac{1-m}{k{y}_i+1}-d-c\left(1-m\right)y_i}& 0\\ 0& {\displaystyle e{y}_i\left(\frac{n}{{(y_i+1)}^2}-f\right)}\end{array}\right),$$

(14)

the two eigenvalues of $J\left(E_i\right)$ are expressed as follows:

$${\displaystyle {\lambda }_1=m+\frac{1-m}{k{y}_i+1}-d-c\left(1-m\right)y_i,{\lambda }_2=e{y}_i\left(\frac{n}{{(y_i+1)}^2}-f\right).}$$

From the equation in (8), we have

$$n=-f{y}^2-fy+y+1$$

and

$$F^{\prime }\left(y\right)=2fy+f-1.$$

Then, ${\lambda }_2$ and $F^{\prime }\left(y\right)$ can be simplified as follows:

$${\lambda }_2=-{\displaystyle \frac{e{y}_i\left(2f{y}_i+f-1\right)}{y_i+1}},$$

$$F^{\prime }\left(y_i\right)=2f{y}_i+f-1.$$

Thus, it is easy to obtain that

$${\lambda }_2=-{\displaystyle \frac{e{y}_i}{y_i+1}}{F}^{\prime }\left(y_i\right)$$

(1) For $E_1(0,y_1)$, $y_1={\displaystyle \frac{1-f+\sqrt{{\Delta }_1}}{2f}}$, it can be easily obtained that ${\lambda }_2<0$. Consequently, if $m>g\left(y_1\right)$ i.e.,${\lambda }_1>0$, then $E_1$ is a saddle; if $m<g\left(y_1\right)$ i.e.,${\lambda }_1<0$, then $E_1$ is a stable node. If $m=g\left(y_1\right)$, implying ${\lambda }_1=0$, we proceed with the following analysis. By applying the transformation $(X,Y)=(x,y-y_1)$ and expanding the system (5) around the origin using Taylor’s series, we obtain the following:

$$\left\{\begin{array}{c}{\displaystyle \frac{dX}{dt}=a_{20}{X}^2+a_{11}XY+O\left({\left|X,Y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dY}{dt}=b_{01}Y+b_{02}{Y}^2+O\left({\left|X,Y\right|}^2\right),}\hfill \end{array}\right.$$

(15)

where

$$\begin{array}{cc}& a_{20}=-1,a_{11}={\displaystyle \frac{(c{k}^2{y}_1^2+2ck{y}_1+k+c)(g\left(y_1\right)-1)}{{(k{y}_1+1)}^2}},\hfill \\ & b_{01}=e{y}_1\left({\displaystyle \frac{n}{{(y_1+1)}^2}}-f\right),b_{02}=-{\displaystyle \frac{(f+3f{y}_1+3f{y}_1^2+f{y}_1^3-n)e}{{(1+y_1)}^2}}.\hfill \end{array}$$

Let $b_{01}t=\tau$ ($b_{01}<0$); then, system (15) becomes

$$\left\{\begin{array}{c}{\displaystyle \frac{dX}{d\tau }=\frac{a_{20}}{b_{01}}{X}^2+\frac{a_{11}}{b_{01}}XY+O\left({\left|X,Y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dY}{d\tau }=Y+\frac{b_{02}}{b_{01}}{Y}^2+O\left({\left|X,Y\right|}^3\right).}\hfill \end{array}\right.$$

(16)

Utilizing the central manifold theorem, with the assumption $Y=m{X}^2+n{X}^3+{O\left(\right|X|}^3)$, substitute this expression into the second equation of the system (16). We can achieve $Y=0$. The constrained system limited to the central manifold is expressed as follows:

$${\displaystyle \frac{dX}{d\tau }=\frac{a_{20}}{b_{01}}{X}^2+{O\left(\right|X|}^2).}$$

We know that ${\displaystyle \frac{a_{20}}{b_{01}}}=-{\displaystyle \frac{1}{{\lambda }_2}}>0$. Consequently, invoking Theorem 7.1 in Zhang et al. [39] shows that $E_1$ is an attracting saddle-node. In this configuration, the parabolic sector is positioned to the right of the y-axis, with the two hyperbolic sectors situated in the second and third quadrants, respectively.

(2) For $E_2(0,y_2)$, $y_2={\displaystyle \frac{1-f-\sqrt{{\Delta }_1}}{2f}}$, it can be easily obtained that ${\lambda }_2>0.$ Similarly, if $m>g\left(y_2\right)$, then $E_2$ is an unstable node; if $m<g\left(y_2\right)$, then $E_2$ is a saddle; if $m=g\left(y_2\right)$, $E_2$ is a repelling saddle-node. This configuration positions the parabolic sector to the left of the y-axis, with the hyperbolic sectors residing in the first and fourth quadrants.

(3) For $E_3(0,y_3)$, $y_3={\displaystyle \frac{1-f^{-}}{2f^{-}}}$, it can be easily obtained that ${\lambda }_2=0.$ Apparently, if $m\ne g\left(y_3\right)$, i.e., ${\lambda }_1\ne 0$, we proceed with the ensuing analysis by introducing the coordinate transformation $(X,Y)=(x,y-y_3)$ and executing a Taylor-series expansion centered at the origin, system (5) becomes

$$\left\{\begin{array}{c}{\displaystyle \frac{dX}{dt}=c_{10}X+c_{20}{X}^2+c_{11}XY+O\left({\left|X,Y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dY}{dt}=d_{02}{Y}^2+O\left({\left|X,Y\right|}^2\right),}\hfill \end{array}\right.$$

(17)

where

$$\begin{array}{cc}& {\displaystyle c_{10}=m+\frac{1-m}{k{y}_3+1}-d-c\left(1-m\right)y_3,c_{20}=-1,c_{11}=\frac{\left(-1+m\right)\left(c{k}^2{{\mathit{y}}_{\mathit{3}}}^2+2ck{\mathit{y}}_{\mathit{3}}+c+k\right)}{{\left(k{\mathit{y}}_{\mathit{3}}+1\right)}^2},}\hfill \\ & d_{02}=-{\displaystyle \frac{e\left(f^{-}{{\mathit{y}}_{\mathit{3}}}^3+3f^{-}{{\mathit{y}}_{\mathit{3}}}^2+3f^{-}{\mathit{y}}_{\mathit{3}}+f^{-}-n\right)}{{\left({\mathit{y}}_{\mathit{3}}+1\right)}^3}}.\hfill \end{array}$$

Let $c_{10}t=\tau$, and then system (17) becomes

$$\left\{\begin{array}{c}{\displaystyle \frac{dX}{d\tau }=X+\frac{c_{20}}{c_{10}}{X}^2+\frac{c_{11}}{c_{10}}XY+O\left({\left|X,Y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dY}{d\tau }=\frac{d_{02}}{c_{10}}{Y}^2+O\left({\left|X,Y\right|}^2\right).}\hfill \end{array}\right.$$

(18)

For computational simplicity, when $f=f^{-}$, we solve for $n={\displaystyle \frac{{(f^{-}+1)}^2}{4f^{-}}}$. Substituting $y_3={\displaystyle \frac{1-f^{-}}{2f^{-}}}$ and $n={\displaystyle \frac{{(f^{-}+1)}^2}{4f^{-}}}$ into a yields $d_{02}={\displaystyle \frac{e{f}^{-}\left(f^{-}-1\right)}{f^{-}+1}}<0$. Thus, we know that $d_{02}<0$, and then, when $m>g\left(y_3\right)$, ${\displaystyle \frac{d_{02}}{c_{10}}}<0$. According to Theorem 7.1 from Zhang et al. [39] and a subsequent time transformation analysis, $E_3$ is a repelling saddle-node, and the parabolic sector lies below the x-axis, while the two hyperbolic sectors are situated in the first and second quadrants, respectively. When $m<g\left(y_3\right)$, then ${\displaystyle \frac{d_{02}}{c_{10}}}>0$; it shows that $E_3$ is an attracting saddle-node, with its parabolic sector now situated above the x-axis and the hyperbolic sectors occupying the third and fourth quadrants. When $m=g\left(y_3\right)$, system (17) becomes

$$\left\{\begin{array}{c}{\displaystyle \frac{dX}{dt}=c_{20}{X}^2+c_{11}XY+O\left({\left|X,Y\right|}^2\right),}\hfill \\ {\displaystyle \frac{dY}{dt}=d_{02}{Y}^2+O\left({\left|X,Y\right|}^2\right).}\hfill \end{array}\right.$$

(19)

Utilizing the central manifold theorem, we can achieve $Y=0$. The constrained system, limited to the central manifold is expressed as follows:

$${\displaystyle \frac{dX}{dt}=-X^2+{O\left(\right|X|}^2).}$$

Hence, Theorem 7.1 in Zhang et al. [39] shows that $E_3$ is a saddle.

The proof of Theorem 4 is completed. □

Theorem 5.

The stability of the positive equilibria for system (5) is shown below:

(1) 

For $E_1^{*}(x_1^{*},y_1)$, when $E_1^{*}$ exists, $E_1^{*}$ is a stable node.

(2) 

For $E_2^{*}(x_2^{*},y_2)$, when $E_2^{*}$ exists, $E_2^{*}$ is a saddle.

(3) 

For $E_3^{*}(x_3^{*},y_3)$, when $E_3^{*}$ exists, $E_3^{*}$ is an attracting saddle-node.

Proof. 

The Jacobian matrix at $E_i^{*}(x_i^{*},y_i)$ is

$$J\left(E_i\right)=\left(\begin{array}{cc}-x_i^{*}& 0\\ 0& {\displaystyle e{y}_i\left(\frac{n}{{(y_i+1)}^2}-f\right)}\end{array}\right),$$

(20)

the two eigenvalues of $J\left(E_i^{*}\right)$ are expressed as follows:

$${\displaystyle {\lambda }_1=-x_i^{*}<0,{\lambda }_2=e{y}_i\left(\frac{n}{{(y_i+1)}^2}-f\right).}$$

It is evident that Theorem 5 can undergo the same analysis as Theorem 4 presented above. Consequently, for the sake of brevity, the proof of this theorem is not included here.

The proof of Theorem 5 is completed.  □

We present the potential equilibria and their stability across different parameter ranges in system (5) through Figure 3, Figure 4 and Figure 5 to validate the accuracy of Theorems 2–5. All phase portraits were generated using the MATLAB 2012a tool pplane [40].

Table 2. Stationary states and their stability in system (5).

Equilibrium	Existence		Type
$E_0(0,0)$	Always exists		$0<n<1$, Unstable node
			$n\ge 1$, Saddle
$E_d(1-d,0)$	Always exists		$0<n<1$, Saddle
			$n=1,f\ne 1$, Saddle-node
			$n=1,f=1$, Unstable node
			$n>1$, Stable node
$E_1(0,y_1)$	$0<n<1$		$m>g\left(y_1\right)$, Saddle
	$n=1,f<1$		$m<g\left(y_1\right)$, Stable node
	$n>1,0<f<f^{-}$		$m=g\left(y_1\right)$, Saddle-node
$E_2(0,y_2)$	$n>1,0<f<f^{-}$		$m>g\left(y_2\right)$, Unstable node
			$m<g\left(y_2\right)$, Saddle
			$m=g\left(y_2\right)$, Saddle-node
$E_3(0,y_3)$	$n>1,f=f^{-}$		$m\ne g\left(y_3\right)$, Saddle-node
			$m=g\left(y_3\right)$, Saddle
$E_1^{*}(x_1^{*},y_1)$	$0<n<1$	$m>g\left(y_1\right)$	Stable node
	$n=1,f<1$
	$n>1,0<f<f^{-}$
$E_2^{*}(x_2^{*},y_2)$	$n>1,0<f<f^{-}$	$m>g\left(y_2\right)$	Saddle
$E_3^{*}(x_3^{*},y_3)$	$n>1,f=f^{-}$	$m>g\left(y_3\right)$	Saddle-node

delineates the local dynamical characteristics of the equilibria of system (5).

3.3. Bifurcation Analysis

3.3.1. Transcritical Bifurcation

Based on Theorems 2 and 3, an intriguing occurrence is noted: when $n=1$, the boundary equilibrium $E_0$ of system (5) coincides with $E_1\left(E_2\right)$. And when $n<1$, $E_0$ is an unstable node; when $n>1$, $E_0$ is a saddle, indicating that the stability of $E_0$ changes at $n=1$. This transpires as a result of a transcritical bifurcation at $E_0$.

Theorem 6.

System (5) experiences a transcritical bifurcation around $E_0$ if $n\equiv n_{TC}=1$.

Proof. 

When $n=1$, the Jacobian matrix at $E_0$ is

$$J_{E_0}(0,0;n_{TC})=\left(\begin{array}{cc}1-d& 0\\ 0& 0\end{array}\right).$$

Obviously, $Det\left(J_{E_0}\right)=0,$ so $J_{E_0}$ exists an eigenvalue ${\lambda }_2=0$. Let v and u denote the eigenvectors associated with the eigenvalue ${\lambda }_2=0$ of $J_{E_0}$ and $J_{E_0}^T$, respectively. We can obtain that

$$v=\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}0\\ 1\end{array}\right),w=\left(\begin{array}{c}{w}_1\\ w_2\end{array}\right)=\left(\begin{array}{c}0\\ 1\end{array}\right).$$

(21)

Denote

$$Q=\left(\begin{array}{c}F\\ G\end{array}\right)=\left({\displaystyle \begin{array}{c}{\displaystyle x\left(m+\frac{1-m}{1+ky}-d-x-c(1-m)y\right)}\\ {\displaystyle ey\left(1-fy-\frac{n}{y+1}\right)}\end{array}}\right)·$$

(22)

Furthermore, we have

$$Q_n(E_0;n_{TC})=\left(\begin{array}{c}0\\ 0\end{array}\right),$$

(23)

$$D{Q}_n(E_0;n_{TC})v=\left(\begin{array}{cc}0& 0\\ 0& {\displaystyle -e}\end{array}\right)\left(\begin{array}{c}0\\ 1\end{array}\right)=\left(\begin{array}{c}0\\ \\ -e\end{array}\right),$$

(24)

$$\begin{array}{cc}\hfill D^{2}Q(E_0;n_{TC})(v,v)& ={\left({\displaystyle \genfrac{}{}{0ex}{}{\frac{{\partial }^{2}F}{\partial x^2}{v}_1^2+2\frac{{\partial }^{2}F}{\partial x\partial y}{v}_1{v}_2+\frac{{\partial }^{2}F}{\partial y^2}{v}_2^2}{\frac{{\partial }^{2}G}{\partial x^2}{v}_1^2+2\frac{{\partial }^{2}G}{\partial x\partial y}{v}_1{v}_2+\frac{{\partial }^{2}G}{\partial y^2}{v}_2^2}}\right)}_{(E_0;n_{TC})}\hfill \\ & ={\left({\displaystyle \genfrac{}{}{0ex}{}{0}{2e\left(-f+\frac{n}{{\left(y+1\right)}^2}\right)-2\frac{eyn}{{\left(y+1\right)}^3}}}\right)}_{(E_0;n_{TC})}\\ & =\left(\genfrac{}{}{0ex}{}{0}{2e(1-f)}\right)·\end{array}$$

(25)

Since $f\ne 1$, it follows from (23)–(25) that

$$\begin{array}{cc}\hfill & w^T{Q}_n(E_0;n_{TC})=0,\hfill \\ \hfill & {\displaystyle w^T\left[D{Q}_n(E_0;n_{TC})v\right]=-e\ne 0,}\hfill \\ \hfill & {\displaystyle w^T\left[D^{2}Q(E_0;n_{TC})(v,v)\right]=2e(1-f)\ne 0.}\hfill \end{array}$$

(26)

In accordance with Sotomayor’s Theorem [41], system (5) experiences a transcritical bifurcation near $E_0$ when $n=n_{TC}$.

The proof of Theorem 6 is completed.  □

Theorem 7.

When $n>1$ and $0<f<f^{-}$, system (5) experiences a transcritical bifurcation around $E_1$ if $m\equiv m_{TC}=g\left(y_1\right)$.

Proof. 

The proof of this theorem is similar to that of Theorem 6; thus, we do not present the detailed proof here.  □

3.3.2. Pitchfork Bifurcation

According to Theorem 6, when $f=1$, the third transversality condition for the transcritical bifurcation at $E_0$ is given by (26) equals to zero. Furthermore, when $n=n_{TC}$ and $f=1$, $E_0$ becomes a saddle. With $f=1$ held constant, as the parameter n varies, $E_1$ and $E_2$ will move to coincide with $E_0$. This occurs due to a pitchfork bifurcation at the boundary equilibrium $E_0$.

Theorem 8.

System (5) experiences a pitchfork bifurcation around $E_0$ when $n=n_{TC}$ and $f=1$.

Proof. 

From Theorem 6, we have

$$w^T{Q}_n(E_0;n_{TC})=0,$$

$$w^T\left[Q_n(E_0;n_{TC})v\right]=-e\ne 0,$$

when $\eta ={\eta }^{*}$, we get

$${\displaystyle w^T\left[D^{2}Q(E_0;n_{TC})(v,v)\right]=2e(1-f)=0.}$$

Through simple calculation, we have

$$\begin{array}{cc}\hfill D^{3}Q(E_0;n_{TC})(v,v,v)& ={\left({\displaystyle \genfrac{}{}{0ex}{}{\frac{{\partial }^{3}F}{\partial x^3}{v}_1^3+\frac{{\partial }^{3}F}{\partial x^2\partial y}{v}_1^2{v}_2+\frac{{\partial }^{3}F}{\partial x\partial y^2}{v}_1{v}_2^2+\frac{{\partial }^{3}F}{\partial y^3}{v}_2^3}{\frac{{\partial }^{3}G}{\partial x^3}{v}_1^3+\frac{{\partial }^{3}G}{\partial x^2\partial y}{v}_1^2{v}_2+\frac{{\partial }^{3}G}{\partial x\partial y^2}{v}_1{v}_2^2+\frac{{\partial }^{3}G}{\partial y^3}{v}_2^3}}\right)}_{(E_0;n_{TC})}\hfill \\ & =\left(\genfrac{}{}{0ex}{}{0}{-6e}\right)·\end{array}$$

(27)

Obviously,

$${\displaystyle w^T\left[D^{3}Q(E_0;n_{TC})(v,v,v)\right]=-6e\ne 0.}$$

Consequently, in line with Sotomayor’s theorem [41], when $n=n_{TC}$ and $f=1$, system (5) experiences a pitchfork bifurcation at $E_0$.

The proof of Theorem 8 is completed.  □

3.3.3. Saddle-Node Bifurcation

Based on Theorem 2, when $n>1$, the system (5) exhibits no positive equilibria for $f>f^{-}$, one positive equilibrium, $E_3$, for $f=f^{-}$, and two positive equilibria, $E_1$ and $E_2$, for $0<f<f^{-}$. Thus, we determine that a saddle-node bifurcation takes place at $E_3$ in system (5).

Theorem 9.

When $n>1$, system (5) experiences a saddle-node bifurcation around $E_3$ with respect to the parameter f if $f=f^{-}$.

Proof. 

When $f=f^{-}$, the Jacobian matrix at $E_3$ is

$$J_{E_3}(0,y_3;f^{-})=\left(\begin{array}{cc}{\displaystyle m+\frac{1-m}{k{y}_3+1}-d-c\left(1-m\right)y_3}& 0\\ 0& 0\end{array}\right).$$

Obviously, $Det\left(J_{E_3}\right)=0,$ indicating that $J_{E_3}$ possesses a zero eigenvalue ${\lambda }_2=0$. Define V and W as the eigenvectors for ${\lambda }_2=0$ in $J_{E_3}$ and $J_{E_3}^T$, respectively. Then, they can be given by

$$V=\left(\begin{array}{c}{V}_1\\ V_2\end{array}\right)=\left(\begin{array}{c}0\\ 1\end{array}\right),W=\left(\begin{array}{c}{W}_1\\ W_2\end{array}\right)=\left(\begin{array}{c}0\\ 1\end{array}\right).$$

(28)

Furthermore, we can get

$$Q_f(E_3;f^{-})={\left({\displaystyle \genfrac{}{}{0ex}{}{\frac{\partial F}{\partial f}}{\frac{\partial G}{\partial f}}}\right)}_{(E_3;f^{-})}=\left(\begin{array}{c}0\\ {\displaystyle -e{y}_3^2}\end{array}\right),$$

(29)

$$D^{2}Q(E_3;f^{-})(V,V)={\left({\displaystyle \genfrac{}{}{0ex}{}{\frac{{\partial }^{2}F}{\partial x^2}{V}_1^2+2\frac{{\partial }^{2}F}{\partial x\partial y}{V}_1{V}_2+\frac{{\partial }^{2}F}{\partial y^2}{V}_2^2}{\frac{{\partial }^{2}G}{\partial x^2}{V}_1^2+2\frac{{\partial }^{2}G}{\partial x\partial y}{V}_1{V}_2+\frac{{\partial }^{2}G}{\partial y^2}{V}_2^2}}\right)}_{(E_3;f^{-})}=\left(\genfrac{}{}{0ex}{}{0}{N}\right),$$

(30)

where $N=-{\displaystyle \frac{2e\alpha }{{\left(1+y_3\right)}^3}}$ with $\alpha =-n+{\left(1+y_3\right)}^3{f}^{-}.$

Since whether N equals zero is determined by $\alpha$, for computational simplicity, when $f=f^{-}$, we solve for $n={\displaystyle \frac{{(f^{-}+1)}^2}{4f^{-}}}$. Substituting $y_3={\displaystyle \frac{1-f^{-}}{2f^{-}}}$ and $n={\displaystyle \frac{{(f^{-}+1)}^2}{4f^{-}}}$ into a yields

$$\alpha =-{\displaystyle \frac{{\left(f^{-}+1\right)}^2\left(f^{-}-1\right)}{8{f^{-}}^2}}\ne 0$$

since $f^{-}\ne 1$, thereby proving $N\ne 0$.

It follows from (29) and (30) that

$${\displaystyle W^T{Q}_f(E_3;f^{-})=-e{y}_3^2\ne 0,}$$

$$W^T\left[D^{2}Q(E_3;f^{-})(V,V)\right]=N\ne 0.$$

Therefore, according to Sotomayor’s Theorem [41], system (5) experiences a saddle-node bifurcation around $E_3$ at $f=f^{-}$.

The proof of Theorem 9 is completed.  □

When $n>1$ and $m>max\left\{g\left(y_1\right),g\left(y_2\right),g\left(y_3\right)\right\}$, the system (5) exhibits no positive equilibria for $f>f^{-}$, one positive equilibrium $E_3^{*}$ for $f=f^{-}$, and two positive equilibria $E_1^{*}$ and $E_2^{*}$ for $0<f<f^{-}$. Thus, we determine that a saddle-node bifurcation takes place at $E_3^{*}$ in system (5).

Theorem 10.

When $n>1$ and $m>max\left\{g\left(y_1\right),g\left(y_2\right),g\left(y_3\right)\right\}$, system (5) experiences a saddle-node bifurcation around $E_3^{*}$ with respect to the parameter f if $f=f^{-}$.

Proof. 

The proof of this theorem is similar to that of Theorem 9; thus, we do not present the detailed proof here.  □

3.4. Global Structure

To understand the overall behavior of system (5) and gain insights into the trajectory patterns as $\left|x\right|+\left|y\right|\to \infty$, we will examine the stability at infinity by applying the Poincaré transformation, as outlined in Chapter 5 of the reference Zhang et al. [39]:

$${\displaystyle x=\frac{v}{z},y=\frac{1}{z},ds=\frac{dt}{z},}$$

system (5) can be transformed into

$$\left\{\begin{array}{c}{\displaystyle \frac{dv}{ds}=zv\left(m-d-e+\frac{(1-m)z}{z+k}+\frac{nz}{1+z}\right)+v(ef-v-c+cm),}\hfill \\ {\displaystyle \frac{dz}{ds}=-ze\left(z-f-\frac{n{z}^2}{1+z}\right).}\hfill \end{array}\right.$$

(31)

With $z=0$, system (31) exists two equilibria $D_1(0,0)$ and $D_2(f+cm-c,0)$ if $m>{\displaystyle \frac{c-f}{c}}$ and one equilibrium $D_1(0,0)$ if $m\le {\displaystyle \frac{c-f}{c}}$ in the first quadrant of $v-z$ plane.

Next, we apply the second type of Poincaré transformation:

$${\displaystyle x=\frac{1}{z},y=\frac{u}{z},ds=\frac{dt}{z},}$$

system (5) becomes

$$\left\{\begin{array}{c}{\displaystyle \frac{du}{ds}=-uz\left(m-d-e+\frac{(1-m)z}{z+ku}+\frac{nz}{u+z}\right)+u+(c-cm-f)u^2,}\hfill \\ {\displaystyle \frac{dz}{ds}=-z^2\left(m-d+\frac{(1-m)z}{z+ku}\right)+z+c(1-m)zu.}\hfill \end{array}\right.$$

(32)

Obviously, in system (32) exists one boundary equilibrium $D_3({\displaystyle \frac{1}{f+cm-c}},0)$ if $m>{\displaystyle \frac{c-f}{c}}$ and no boundary equilibria if $m\le {\displaystyle \frac{c-f}{c}}$ in the nonnegative cone of $u-z$ plane.

Theorem 11.

In system (31), $D_1(0,0)$ is an unstable node, and $D_2(f+cm-c,0)$ is a saddle. In system (32), $D_3({\displaystyle \frac{1}{cm+f-c}},0)$ is a saddle.

Proof. 

The Jacobian matrix of system (31) at $D_1$ and $D_2$ are

$$J\left(D_1\right)=\left(\begin{array}{cc}f+cm-c& 0\\ 0& f\end{array}\right)$$

(33)

and

$$J\left(D_2\right)=\left(\begin{array}{cc}c-f-cm& -(-c+f+cm)(d+e-m)\\ 0& f\end{array}\right).$$

(34)

Obviously, $D_1$ is an unstable node, and $D_2$ is a saddle.

The Jacobian matrix of system (32) at $D_3$ is

$$J\left(D_3\right)=\left(\begin{array}{cc}-1& {\displaystyle \frac{d+e-m}{-c+f+cm}}\\ 0& {\displaystyle \frac{f}{-c+f+cm}}\end{array}\right).$$

(35)

It is easy to confirm that $D_3$ is a saddle.

The proof of Theorem 11 is completed. □

According to the Poincaré transformation perspective, we obtain

$$x_1\triangleq {\displaystyle \frac{v}{z}}=0,y_1\triangleq {\displaystyle \frac{1}{z}}=+\infty .$$

$$x_2\triangleq {\displaystyle \frac{v}{z}}=+\infty ,y_2\triangleq {\displaystyle \frac{1}{z}}=+\infty .$$

$$x_3\triangleq {\displaystyle \frac{1}{z}}=+\infty ,y_3\triangleq {\displaystyle \frac{u}{z}}=+\infty ,$$

Consequently, $D_1$ represents the infinity point $I_1$ in system (5), which corresponds with the critical point at infinity of the y-axis and is an unstable node. $D_2$ and $D_3$ represent the infinity point I, which corresponds with the critical point at infinity and is a saddle. Figure 6 displays the Poincaré compactification of system (5), revealing its dynamics at infinity.

Next, we prove whether system (5) has closed orbits. We partition the parameter space $(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}$ into five distinct regions:

$$\begin{array}{cc}\hfill R_1=& R_{11}\cup R_{12}\cup R_{13}\cup R_{14}\cup R_{15}\cup R_{16};\hfill \\ \hfill R_2=& R_{21}\cup R_{22};\hfill \\ \hfill R_3=& \{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,0<f<f^{-},g\left(y_2\right)<m\le g\left(y_1\right)\};\hfill \\ \hfill R_4=& \{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,0<f<f^{-},m>g\left(y_1\right)\};\hfill \\ \hfill R_5=& \{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,f=f^{-},m>g\left(y_3\right)\}.\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill R_{11}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:0<n<1,m\le g\left(y_1\right)\};\hfill \\ \hfill R_{12}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n=1,f<1,m\le g\left(y_1\right)\};\hfill \\ \hfill R_{13}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,0<f<f^{-},m\le g\left(y_2\right)\};\hfill \\ \hfill R_{14}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,f=f^{-},m\le g\left(y_3\right)\};\hfill \\ \hfill R_{15}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n=1,f\ge 1\};\hfill \\ \hfill R_{16}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n>1,f>f^{-}\};\hfill \\ \hfill R_{21}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:0<n<1,m>g\left(y_1\right)\};\hfill \\ \hfill R_{22}& =\{(c,d,e,f,k,m,n)\in {\mathbb{R}}_7^{+}:n=1,f<1,m>g\left(y_1\right)\}.\hfill \end{array}$$

Theorem 12.

The system (5) has no closed orbit.

Proof. 

Within the parameter space $(c,d,e,f,k,m,n)\in R_1$, system (5) lacks internal equilibrium points, precluding the possibility of closed orbits. The invariance of the lines $x=0,y=0$ and $y=y_1$ ensures that any closed path would intersect these, contradicting their defining property.

Next, we examine the case where $(c,d,e,f,k,m,n)\in R_2$. We can assert that no closed orbits exist in $R_2$. Suppose, for contradiction, that system (5) admits a closed orbit. Based on Theorem 4.6 in Chapter 4 of Zhang et al. [39], the interior of such a closed orbit must contain an equilibrium point. Yet, given that $y=y_1$ represents an invariant manifold and the internal equilibrium $E_1^{*}$ is positioned on this particular curve, the supposed closed orbit would necessarily cross $y=y_1$ at two separate locations. This scenario directly conflicts with the fundamental theorem governing the uniqueness of solutions.

A similar argument applies to the cases where $(c,d,e,f,k,m,n)\in R_3,R_4$ or $R_5$, as $y=y_2$ and $y=y_3$ are also invariant sets. Thus, according to analogous reasoning, system (5) cannot exhibit closed orbits in these parameter regimes either.

The proof of Theorem 12 is completed.  □

Theorem 13.

(i) When $n=1$, $f\ge 1$ or $n>1$, $f>f^{-}$ or $n=1$, $f=f^{-}$, the boundary equilibrium $E_d$ is globally asymptotically stable; (ii) when $0<n<1$, $m<g\left(y_1\right)$ or $n=1,$$f<1$ and $m<g\left(y_1\right)$, the boundary equilibrium $E_1$ is globally asymptotically stable; (iii) when $0<n<1$, $m>g\left(y_1\right)$ or $n=1$, $f<1$ and $m>g\left(y_1\right)$, the positive equilibrium $E_1^{*}$ is globally asymptotically stable.

Proof. 

From Table 2, we found that, when $n=1$, $f\ge 1$ or $n>1$, $f>f^{-}$, system (5) has only two boundary equilibria $E_0$ and $E_d$, of which $E_0$ is unstable and $E_d$ is locally stable. When $n>1$ and $f=f^{-}$, system (5) may have three boundary equilibria, $E_0$, $E_d$, and $E_3$, and one positive equilibrium, $E_3^{*}$, in which $E_0$, $E_3$, and $E_3^{*}$ are all unstable, and only $E_d$ is locally stable. Since Theorem 12 confirms the absence of closed orbits in the system (5), $E_d$ is consequently globally asymptotically stable. The analysis of global asymptotic stability for both $E_1$ and $E_1^{*}$ in system (5) follows a similar approach to that of $E_d$, and thus, the detailed proof is omitted here for brevity.  □

Theorem 14.

(i) When $n>1,0<f<f^{-}$ and $m<g\left(y_1\right)$, the stable manifolds associated with $E_2^{*}$ partition the first quadrant into distinct attraction regions corresponding to $E_1$ and $E_d$, creating system (5) bistability; (ii) when $0<n<1,0<f<f^{-}$ and $m>g\left(y_1\right)$, the stable manifolds associated with $E_2^{*}$ partition the first quadrant into distinct attraction regions corresponding to $E_1^{*}$ and $E_d$, creating system (5) bistability.

Proof. 

When $n>1,0<f<f^{-}$ and $m<g\left(y_1\right)$, system (5) may have four boundary equilibria, $E_0$, $E_d$, $E_1$, and $E_2$, and one positive equilibrium, $E_2^{*}$, of which $E_0$, $E_2$, and $E_2^{*}$ are all unstable, and only $E_d$ and $E_1$ are locally stable. As shown in Figure 4b, the stable manifold corresponding to $E_2^{*}$ forms the separating boundary that distinguishes the attraction regions of $E_1$ and $E_d$ in system (5). This indicates that the system’s asymptotic behavior depends on initial conditions: trajectories starting within the basin of attraction of $E_1$ ($E_d$) will ultimately converge to $E_1$($E_d$), respectively. When the refuge proportion m exceeds the threshold $g\left(y_1\right)$, the system (5) undergoes a bifurcation where $E_1^{*}$ emerges and replaces $E_1$ as a stable node. Figure 4c demonstrates the coexistence of two stable states-the positive equilibrium $E_1^{*}$ and the boundary equilibrium $E_d$. Biologically, the threshold $g\left(y_1\right)$ represents the minimum refuge requirement for maintaining biodiversity. With insufficient refuge ($m<g\left(y_1\right))$, the first population inevitably faces extinction. However, when m surpasses the critical value $g\left(y_1\right)$, the emergence of the stable state $E_1^{*}$ enables the potential coexistence of both populations. This demonstrates that the refuge parameter m is a crucial determinant of whether the ecosystem achieves species coexistence or collapses into single-species dominance, with $g\left(y_1\right)$ marking a critical ecological threshold in the system’s dynamics.  □

3.5. Sensitivity Analysis

In ecology and mathematical biology, sensitivity analysis is a crucial tool for studying the impact of system parameters on model outputs. Through sensitivity analysis, we can quantify the influence of system parameters on population dynamics, identify key parameters, and provide a basis for model simplification and optimization [42,43,44]. In this section, we will perform semi-relative and logarithmic sensitivity analyses on system (5) to evaluate the effects of various parameters on population dynamics.

The semi-relative sensitivity function $S_{i,j}$ is defined as follows:

$$S_{i,j}=q_j{\displaystyle \frac{\partial u_i}{\partial q_j}},$$

where $u_i$ is the response variable (x or y), $q_j$ is a system parameter ($m,d,k,e,f,n,a$), and ${\displaystyle \frac{\partial u_i}{\partial q_j}}$ is the partial derivative of the response variable $u_i$ with respect to the parameter $q_j$. The logarithmic sensitivity function ${\widehat{S}}_{i,j}$ is defined as follows:

$${\widehat{S}}_{i,j}={\displaystyle \frac{q_j}{u_i}}{\displaystyle \frac{\partial u_i}{\partial q_j}}.$$

The logarithmic sensitivity function measures the percentage change in the response variable $u_i$ due to a unit percentage change in the parameter $q_j$.

To compute the partial derivative ${\displaystyle \frac{\partial u_i}{\partial q_j}}$, we employ the central difference method by perturbing the parameter $q_j$ by a small amount $\Delta q_j$ (typically no more than 10% of $q_j$), numerically solving system (5) to obtain $u_i(q_j+\Delta q_j)$ and $u_i(q_j-\Delta q_j)$, and then applying the central difference formula [45]:

$${\displaystyle \frac{\partial u_i}{\partial q_j}}\approx {\displaystyle \frac{u_i(q_j+\Delta q_j)-u_i(q_j-\Delta q_j)}{2\Delta q_j}}.$$

Using MATLAB numerical simulations, we obtained the semi-relative sensitivity and logarithmic sensitivity of the stable coexistence equilibrium $E_1^{*}$ with respect to the first population x and the second population y, as shown in Figure 7. We observe that the density of the first population is highly sensitive to the refuge parameter m and the mortality rate d of the first population. Specifically, the refuge parameter m has a positive effect on the density of the first population, while the mortality rate d of the first population exerts a negative influence. Additionally, the fear level k has a slight negative impact on the density of the first population but no significant effect on the density of the second population. On the other hand, the density of the second population demonstrates high sensitivity to both the Allee effect parameter n and the intraspecific competition coefficient f, with both parameters exerting negative effects. The influence of other factors on its density is comparatively minor, highlighting the crucial role of additive Allee effects in maintaining coexistence equilibrium.

As illustrated in Figure 7a,c, perturbations in the refuge parameter m produce significantly stronger positive impacts on the first population’s density than the negative effects induced by the mortality rate d. This indicates that when sufficient refuge conditions are available, the first population can achieve stable persistence. Notably, the positive effect of the Allee effect parameter n on the first population demonstrates that the synergistic interaction between Allee effects and refuge availability jointly promotes the survival of the first population, thereby facilitating long-term coexistence of the two populations.

Next, we consider the effects of the fear level, k, and the refuge parameter, m, on the system’s dynamical behavior. Let us first consider the influence of refuge on amensalism system without the fear level; i.e., letting $k=0$ in system (5), we have

$${\displaystyle x_i^{*}=1-d-c(1-m)y_i,y_i=\frac{1-f\pm \sqrt{{\Delta }_1}}{2f},}$$

and then

$${\displaystyle \frac{\mathrm{d}{x}_i^{*}}{\mathrm{d}m}}=c{y}_i>0,{\displaystyle \frac{\mathrm{d}{y}_i}{\mathrm{d}m}}=0.$$

Hence, we know that the increase in m can increase the first species, and the density of the second species is unaffected by the fear level, k, and the refuge parameter, m.

Subsequently, we will consider the influence of refuge with fear level on the amensalism system (5), i.e., $k>0$. Similarly, $y_i$ is independent of k and m. Additionally,

$${\displaystyle \frac{\mathrm{d}{x}_i^{*}}{\mathrm{d}m}}=1-{\displaystyle \frac{1}{1+k{y}_i}}+c{y}_i>c{y}_i>0.$$

From a biological perspective, a refuge provides the first species with a space to escape amensal pressure, thereby reducing its mortality rate and enhancing its survival probability. Regardless of the presence of the fear effect, the existence of a refuge positively impacts the survival of the first species. However, when the fear effect is incorporated into the system, the influence of the refuge on the density of the first species becomes more pronounced. This is because the fear effect further restricts the species’ activity range, reduces foraging efficiency, and lowers reproductive success, leading to a further decline in population density. In this context, the refuge not only offers physical protection but also mitigates the behavioral changes induced by the fear effect, enabling the first species to maintain a higher population density. Therefore, in systems with the fear effect, the role of the refuge in supporting the first species is more significant compared to systems without the fear effect. This aligns with the effect of refuges on birth rates as established in the model construction in Section 2.2.

Finally, we investigate the effects of the refuge parameter m and fear level k on the stable coexistence equilibrium under both weak and strong Allee effects. Figure 8 illustrates the synergistic effects of m and k on the density of the first species. The results show that, compared to the strong Allee effect, the weak Allee effect requires a larger refuge parameter m to sustain the survival of the first species, and the species’ density is more sensitive to changes in m. This indicates that, under weak Allee effects, a larger refuge parameter is critical for ensuring species survival. In contrast, under strong Allee effects, while the refuge parameter remains important, its effect is relatively weaker, and even moderate refuge strength can support species survival. Moreover, regardless of whether the Allee effect is weak or strong, an increase in the fear level k necessitates a corresponding increase in the refuge parameter m. That is, a higher fear level demands greater reliance on refuge to maintain species survival.

4. Numerical Simulations

From Theorem 14, it is known that, when $n>1$, $0<f<f^{-}$ and $m>g\left(y_1\right)$, system (5) exhibits bistability, meaning the stable state of the system depends on the initial density of the second species. Depending on the initial density, the system (5) may ultimately stabilize at two different equilibria: $E_d$ and $E_1^{*}$. Similarly, when $n>1$, $0<f<f^{-}$ and $m\le g\left(y_1\right)$, the system also has two stable equilibria: $E_1$ and $E_d$. In the following, we select different initial values for the second species and investigate how the refuge parameter, m, influences the dynamical behavior of the system under certain parameter conditions. We next confirm this using numerical simulation using the time-course graphs of solutions.

Example 1.

For $n>1$ and $0<f<f^{-}$, the case where $(d,k,c,e,f,n)=(0.6,0.5,1,1,0.35,1.2)$ and the initial species densities $\left(x\right(0),y(0\left)\right)=(0.5,0.5)$ is examined. As shown in Figure 9, when $m\le g\left(y_1\right)=0.7884849434$, an increase in refuge parameter m will slow down the extinction process of the first species population; when $m>g\left(y_1\right)=0.7884849434$, as refuge parameter m increases, the density of the first species rises without affecting the species’ ultimate extinction or survival

Example 2.

For $n>1$ and $0<f<f^{-}$, the case where $(d,k,c,e,f,n)=(0.6,0.5,1,1,0.35,1.2)$ and the initial species densities $\left(x\right(0),y(0\left)\right)=(0.5,0.2)$ is examined. As shown in Figure 10, an increase in the refuge parameter m accelerates the time for the first species to reach a stable state, but it has no effect on the second species.

Remark 1.

We observed that, when the second species exhibits a strong Allee effect, i.e., $n>1$, the final densities of both species depend on the initial density of the second species. When the initial density of the second species is low, the system eventually stabilizes at a state where the first species survives, while the second species goes extinct. Additionally, an increase in the refuge parameter m significantly accelerates the time for the first species to reach its stable state. Conversely, when the initial density of the second species is high, and the refuge parameter m is low, i.e., $m\le g\left(y_1\right)$, the system stabilizes with the first species becoming extinct and the second persisting. In this case, raising m decelerates the decline of the first species. When the refuge parameter m is sufficiently high, i.e., $m>g\left(y_1\right)$, both species achieve long-term coexistence, and further increases in m lead to a higher stable density of the first species.

Next, we will illustrate the global stability of the equilibria in Theorem 13 through numerical simulations.

Example 3.

We consider the situation when $(d,k,c,e,f,n,m)=(0.6,0.5,1,1,0.5,1.2,0.5)$. Numerical simulation results (Figure 11 and Figure 12) show that, regardless of the initial values, x eventually tends to 0.4, and y tends to 0, exhibiting that $E_d(0.4,0)$ is globally asymptotically stable.

Example 4.

We examine the situation when $(d,k,c,e,f,n,m)=(0.6,0.5,1,1,0.8,0.7,0.5)$. Numerical simulation results (Figure 13 and Figure 14) show that, regardless of the initial values, x eventually tends to 0, and y tends to $0.75$, exhibiting that $E_1(0,0.75)$ is globally asymptotically stable.

Example 5.

We examine the situation when $(d,k,c,e,f,n,m)=(0.6,0.5,1,1,0.8,0.7,0.8)$. Numerical simulation results (Figure 15 and Figure 16) show that, regardless of the initial values, x eventually tends to $0.1911111113$, and y tends to $0.75$, exhibiting that $E_1^{*}(0.1911111113,0.75)$ is globally asymptotically stable.

Remark 2.

We observe that, if the second species exhibits a weak Allee effect (i.e., $0<n\le 1$), when the refuge parameter, m, is relatively small ($m\le g\left(y_1\right)$), the system (5) has a globally asymptotically stable boundary equilibrium $E_1$, where the first species becomes extinct while the second species persists. At this point, the refuge is not strong enough to support the coexistence of both species. As m increases to a certain threshold ($m>g\left(y_1\right)$), the system (5) transitions to having a single globally asymptotically stable interior equilibrium, $E_1^{*}$, where both species can coexist. This transition indicates that the presence and strength of the refuge not only alleviate the amensal pressure but also, to some extent, enhance the population stability, allowing both species to coexist under certain conditions. Therefore, in the case of a weak Allee effect, the refuge plays a critical role in providing habitat, mitigating amensal pressure, and increasing population stability, which is of significant importance for the long-term survival and coexistence of species.

Remark 3.

When n is chosen as the bifurcation parameter (see Figure 17a,b), the system exhibits a stable boundary equilibrium $E_1$ for relatively small values of n, indicating the extinction of the first species while the second species persists. As n increases, a transcritical bifurcation occurs, leading to the emergence of a stable positive equilibrium, $E_1^{*}$, where both species coexist. When n reaches $n_{SN}$, the two positive equilibria $E_1^{*}$ and $E_2^{*}$ merge into a unique positive equilibrium $E_3^{*}$. Further increasing n results in the absence of positive equilibria, demonstrating that $n<n_{SN}$ is the condition for the existence of positive equilibria. For sufficiently large n, the coexistence of both species becomes impossible, and the system transitions through a saddle-node bifurcation. When $f=1$, $E_1$ and $E_2$ coincide with $E_0$, and the transcritical bifurcation at $E_0$ transitions into a pitchfork bifurcation. When the refuge parameter m is used as the bifurcation parameter (see Figure 17c), the refuge plays a critical role in the survival of the first species. For small values of m, the first species becomes extinct while the second species persists. As m increases, the system undergoes a transcritical bifurcation. The refuge provides the first species with an opportunity to escape the pressure exerted by the second species, causing the positive equilibrium $E_1^{*}$ to collide with the boundary equilibrium, $E_1$, and exchange stability. Ultimately, the system transitions to a state of coexistence between the two species, indicating that the refuge significantly enhances the survival capacity of the first species and drives the system from single-species persistence to coexistence.

Figure 17 was simulated using MATLAB’s MATCONT tool under certain parameter conditions [46].

5. Conclusions

This paper considers an amensalism model where the first species exhibits the refuge and fear effect, while the second species exhibits additive Allee effect. The study investigates the impact of the refuge on the dynamics of the amensalism system under strong and weak Allee effects. We can conclude from theoretical study and numerical simulations.

For the case $0<n<1$ (weak Allee ffect), when the refuge parameter m is small, i.e., $m\le g\left(y_1\right)$, system (5) has a globally asymptotically stable equilibrium $E_1$. As the refuge parameter increases, i.e., $m>g\left(y_1\right)$, system (5) has two equilibria, $E_1$ and $E_1^{*}$, with the positive equilibrium $E_1^{*}$ being globally asymptotically stable. This indicates that the presence of a refuge is beneficial for the long-term survival and coexistence of both species.

For the case $n>1$ (strong Allee effect), when $0<f<f^{-}$, the final densities of both species are determined by the initial density of the second species. When the initial density of the second species is low, due to the additive Allee effect, the second species will eventually go extinct while the first species persists. Additionally, an increase in the refuge parameter m accelerates the convergence of the first species to its stable state. When the initial density of the second species is high, if $m\le g\left(y_1\right)$, the first species will eventually go extinct while the second species persists. In this case, an increase in the refuge parameter, m, slows down the extinction of the first species. However, if $m>g\left(y_1\right)$, both species can coexist in the long term, and increasing the refuge parameter, m, will raise the final density of the first species.

Based on the sensitivity analysis of model parameters, the study reveals that the refuge exerts the most significant positive effect on the survival of the first species, particularly under the presence of the fear effect. When the refuge parameter exceeds a critical threshold, the first species can achieve coexistence with the second species despite the amensalistic pressure imposed by the latter. Moreover, the numerical results indicate that, compared to a strong Allee effect, a weak Allee effect requires a larger refuge size to sustain the survival of the first population.

Compared to Xie’s results [14], the dynamical behavior becomes more complex when the dual effects of the refuge and the fear effect on the first species are considered, along with the additive Allee effect on the second species. By varying the parameters of the additive Allee effect n and the refuge parameter m, fixing $f<1$, system (5) undergoes a transcritical bifurcation at $E_0$ when $n=1$, indicating that the intensity of the Allee effect is one of the key factors determining whether the second species goes extinct. When $n>1$ and $0<f<f^{-}$, system (5) undergoes a transcritical bifurcation at $E_1$ when $m=g\left(y\right)$, which suggests that the refuge parameter m is one of the key factors determining whether both species can coexist. When considering the impact of the Allee effect on refuges, it was found that a weak Allee effect requires a larger refuge to sustain the survival of the first population. When examining the influence of the fear effect on refuges, it was observed that the higher the fear level, the more pronounced the positive impact of refuges on the density of the first species.

This study innovatively integrates three key ecological factors—refuge, the fear effect, and the Allee effect—into an amensalism model, overcoming the limitations of previous studies [14,31] that considered only single factors. This integration more realistically reflects the complexity of species interactions in natural ecosystems, particularly providing theoretical support for analyzing the leaf-surface symbiosis mechanism between T. urticae and P. ulmi [34]. In this study, the refuge not only serves as physical protection but also mitigates the impact of fear effects. Through a critical threshold analysis $(m>g\left(y_1\right))$, the influence of the refuge parameter on species coexistence is clearly defined. A sensitivity analysis reveals that the refuge parameter m has the strongest positive effect on the survival of vulnerable species, offering a prioritized strategy for endangered species conservation. Initial condition analysis enables the model to predict evolutionary outcomes under different population densities, which holds significant implications for conservation planning. In practical applications, adjusting initial densities can promote the extinction of invasive species, or designing refuge structures (e.g., vegetation belts) to ensure $m>g\left(y_1\right)$ can protect native species. Moreover, critical parameter sets $(f^{-},g\left(y_i\right))$ can predict ecological tipping points earlier than traditional population monitoring.

Methodologically, this study has combined theoretical analysis and numerical validation. The theoretical analysis employed Sotomayor’s theorem for bifurcation analysis to reveal critical coexistence conditions, while Poincaré transformation-based global analysis ensured the universality of conclusions. In numerical simulations, MATLAB-generated phase portraits and bifurcation diagrams precisely locate bifurcation points and validate theoretical predictions. Furthermore, this study is the first to apply a dual-index sensitivity analysis (semi-relative and logarithmic sensitivity) in an amensalism system. Compared to the findings of Xie et al. [14], this research uncovered bistability phenomena, identified rich bifurcation behaviors, and analyzed stability at infinity.

However, this study involved certain limitations. The model assumes constant environmental parameters without accounting for stochastic fluctuations in real ecosystems. Additionally, while Osakabe’s experiment [34] revealed stage-dependent variations in amensal effects, the current model does not incorporate this stage-structured aspect of the refuge influence on amensalism systems. Further research is needed to elucidate these potential ecological effects.