Fear Effect on a Predator–Prey Model with Non-Differential Fractional Functional Response

Abstract

In this paper, we study the factor of the fear effect in a predator–prey model with prey refuge and a non-differentiable fractional functional response due to the group defense. Since the functional response is non-differentiable, the dynamics of this system are considerably different from the dynamics of a classical predator–prey system. The persistence, the stability and the existence of the steady states are investigated. We examine the Hopf bifurcation at the unique positive equilibrium. Direct Hopf bifurcation is studied via the central manifold theorem. When the value of the fear factor decreases and is less than a threshold ${\kappa }_H$, the limit cycle appears, and it disappears through a loop of heteroclinic orbits when the value of the fear factor is equal to a value ${\kappa }_{het}$.

1. Introduction

In ecological webs, there are four main types of interactions between species: commensalism, mutualism, predation, and competition [1]. Numerous differential-equation-based systems have been developed to describe the dynamics of these interactions. Among all these four types, predation has received the most attention from academics and has been widely investigated in a variety of scenarios due to the significance and prevalence of predation in the real world. Suppose that the predator and prey densities change continuously with time. The following differential equations represent a generalized predator–prey model containing logistic growth:

$$\begin{array}{cc}\hfill \frac{dz}{dt}& {\displaystyle =\varrho z-\zeta z-\xi z^2-\varphi \left(z\right)s,}\hfill \\ \hfill \frac{ds}{dt}& =\beta \varphi \left(z\right)s-\delta s.\hfill \end{array}$$

(1)

The equations in system (1) depict the dynamics of prey and predator, respectively. The interpretations of z, s, $\varphi \left(z\right)$, $\varrho$, $\xi$, $\beta$, $\zeta$, and $\delta$ are summarized in Table 1. The functional response $\varphi \left(z\right)$ is a major feature in any predator–prey model and it takes different forms depending on the scenario (for example, see [2,3,4,5,6,7,8,9,10,11,12]). In Table 2, we summarize a number of traditional forms of the functional response.

Table 1. A summary of the model parameters and their interpretation.

Symbol	Interpretation	Assumptions
z	Prey density
s	Predator density
$\varphi \left(z\right)$	Number of prey successfully attacked per predator	$\varphi \left(0\right)=0$, ${\varphi }^{\prime }\left(z\right)>1$
$\varrho$	Prey population birth rate
$\zeta$	Rate of natural mortality in prey populations
$\xi$	Mortality rate as a result of interspecies competition
$\kappa$	Level of fear
$\mu$	Capacity of a refuge at t	$\mu \in (0,1)$
$\gamma$	Attack rate per predator and prey
$\beta$	Prey conversion to the predator
$\delta$	Per capita death rate of the predator
$\alpha$	Efficiency of aggregation for prey	$\alpha \in (0,1)$
$\sigma$	Handling time per prey
$z^{\ast }$	Incipient limiting level

Table 2. Several forms of traditional functional responses.

Functional Response Type	$\mathit{\varphi }\left(\mathit{z}\right)$	Reference
type I	$\varphi \left(z\right)=\left\{\begin{array}{cc}\gamma z\hfill & \mathrm{if}z\le z^{\ast },\hfill \\ {\displaystyle \frac{1}{c}}\hfill & \mathrm{if}z>z^{\ast }.\hfill \end{array}\right.$	[13]
type II	${\displaystyle \varphi \left(z\right)=\frac{\gamma z}{1+\gamma \sigma z}.}$	[14]
type III	${\displaystyle \varphi \left(z\right)=\frac{\gamma z^{\theta }}{1+\gamma \sigma z^{\theta }},\theta >1}$.	[15]
type IV	${\displaystyle \varphi \left(z\right)=\frac{\gamma g\left(z\right)}{1+\gamma \sigma g\left(z\right)}}$, different expressions.	[16]

There is a growing belief that the sheer existence of a predator may change the behavior and physiology of prey to the point that it might have an impact on prey populations that is even stronger than direct predation [17,18,19]. According to Cresswell, all animals exhibit a range of anti-predator responses in response to perceived predation danger, including changes in habitat use, foraging behaviors, alertness, and physiological changes [20]. According to Zanette et al. [21], the ability of parents of song sparrows to produce offspring was reduced by 40% merely due to their fear of predators. Field studies demonstrate that the fear effect would lower productivity. Therefore, this factor has drawn the attention of numerous academics [22,23,24,25,26,27,28,29,30]. Thus, we amend system (1) by multiplying the production term by a factor $\psi (\kappa ,s)$ that takes into account the cost of anti-predator defense brought on by fear, resulting in

$$\begin{array}{cc}\hfill \frac{dz}{dt}& {\displaystyle =\varrho \psi (\kappa ,s)z-\zeta z-\xi z^2-\varphi \left(z\right)s,}\hfill \\ \hfill \frac{ds}{dt}& =\beta \varphi \left(z\right)s-\delta s.\hfill \end{array}$$

(2)

According to [31], $\psi (\kappa ,s)$ meets the following conditions:

$$\begin{array}{ccccccc}\hfill & \psi (0,s)=1,\hfill & \hfill & {\displaystyle \underset{\kappa \to \infty }{lim}\psi (\kappa ,s)=0,}\hfill & \hfill & & \hfill \frac{\partial \psi (\kappa ,s)}{\partial \kappa }=0,\\ \hfill & \psi (\kappa ,0)=1,\hfill & \hfill & \underset{s\to \infty }{lim}\psi (\kappa ,s)=0,\hfill & \hfill & & \hfill \frac{\partial \psi (\kappa ,s)}{\partial s}=0.\end{array}$$

(3)

Several functions fulfill the conditions in (3), for example

(i)

${\displaystyle {\psi }_1(k,s)=\frac{1}{1+\kappa s}}$

(ii)

${\psi }_1(k,s)=e^{-\kappa s}$

(iii)

${\displaystyle {\psi }_1(k,s)=\frac{e^{-\kappa s}}{1+\omega sin\kappa s},}$ where $\omega \in (0,1).$

In this paper, we consider ${\displaystyle \psi (\kappa ,s)=\frac{1}{1+\kappa s}}$. On the other hand, refuge can be defined to include any technique employed by prey that minimizes the predation risk. Most researchers have demonstrated that refugia have a stabilizing impact on the prey–predator model. Assume that the capacity of the refugia is $\widehat{\mu }.$ There are two different perspectives on this quantity:

(i)

$\widehat{\mu }=\mu z$, the refuge capacity is proportional to the density of prey;

(ii)

$\widehat{\mu }=\mu$, the refuge capacity is constant.

We modify the functional response to incorporate prey refuges to be a function with respect to $(z-\widehat{\mu })$, where $\widehat{\mu }=\mu z,$ and system (1) becomes as follows:

$$\begin{array}{cc}\hfill \frac{dz}{dt}& {\displaystyle =\frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2-\varphi (z-\mu z)s,}\hfill \\ \hfill \frac{ds}{dt}& =\beta \varphi (z-\mu z)s-\delta s.\hfill \end{array}$$

(4)

In addition, cooperative behavior is widespread among organisms [32], such as safety in numbers (group defense), pack hunting, parental care, animal migration, and clumping. Some animals find safety in numbers by existing in large groups: buffalo live in herds [33], numerous fish species (including tuna) congregate in large schools [34], and geese gather in flocks as they move [35]. Living in a group allows animals to protect themselves. For example, white rhinos and gnus create defensive circles [36]. Ajraldi et al. investigated the group defense technique using $\gamma \sqrt{z}$ as a functional response [37]. After this, a more general functional response $\gamma z^{\alpha }$ to describe the group defense was developed by Venturino and Petrovskii [38], where the “$\alpha$” interpretations are as in Table 1. Depending on Venturino’s functional response, many authors have investigated various scenarios for predator–prey models containing group defense [39,40,41]. For example, the existence and uniqueness of limit cycles and nonexistence of periodic orbits was examined in [42], and a bifurcation analysis of a predator–prey model with cooperative predator hunting and a non-differentiable functional response was investigated by Y. Du et al. [43]. Other researchers took various factors into consideration, such as cannibalism [44], multiplicative noise [45], Leslie–Gower terms [46], the Allee effect [47], prey harvesting [48], and predator harvesting [49]. No author has considered how refuge or fear may affect systems that include Venturino’s group defense. The following system of nonlinear ordinary differential equations provides a model for the interaction between the predator and prey populations with group defense in prey, the fear effect, and prey refuge:

$$\begin{array}{cc}\hfill \frac{dz}{dt}& {\displaystyle =\frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2-\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }},z\ge 0,}\hfill \\ \hfill \frac{ds}{dt}& =\frac{\beta \gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}-\delta s.\hfill \end{array}$$

(5)

For more details, see Figure 1. In addition, Figure 2 shows a graphical representation of the fractional functional response ${\displaystyle \varphi \left(z\right)=\frac{\gamma {(z-\mu z)}^{\alpha }}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}}$ with $\alpha <1$.

Figure 1. Food chain diagram of system (5).

Figure 2. The graphical representation of the functional response. (a) The functional response $\varphi \left(z\right)=\frac{\gamma {(z-\mu z)}^{\alpha }}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}$ has been plotted for different values of $\alpha$ when $\gamma =0.5,$ $\mu =0.6,$ and $\sigma =0.3.$ This figure shows that the value of the functional response ultimately increases (when $z\ge \frac{1}{1-\mu }$) as the efficiency of aggregation for prey increases. (b) The functional response $\varphi \left(z\right)=\frac{\gamma {(z-\mu z)}^{\alpha }}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}$ has been plotted for different values of $\mu$ when $\gamma =0.5, \alpha =0.5,$ and $\sigma =0.3$. This figure shows that the value of the functional response decreases as the refuge capacity increases.

In this paper, we pay particular attention to answering the following question: how do group defense, the fear factor, and the refuge affect the qualitative dynamics of the model? We summarize our findings and the contributions of the paper as follows:

• We consider the type IV functional response, and it is nondifferentiable on the s-axis. The functional response ultimately increases (when $z\ge \frac{1}{1-\mu }$) as the efficiency of aggregation for prey increases and it decreases as the refuge capacity increases.
• The predator population falls into decay if the per capita death rate of the predator is greater than a constant $\theta =\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }$ that depends on several parameters. Note that this $\theta$ decreases as the capacity of a refuge at t increases, and $\theta$ increases (decreases) as the value of the efficiency of aggregation for prey increases if $\frac{(1-\mu )\varrho }{\xi }>1$ ($\frac{(1-\mu )\varrho }{\xi }<1$).
• Because of the $z^{\alpha }$ term, the Jacobian matrix is indeterminate at the origin. Therefore, it is impossible to carry out a stability analysis by merely looking at its eigenvalues. We use the definition of stability to prove that if $\varrho <\zeta$, then $E_0(0,0)$ is stable, and if $\varrho >\zeta$, then $E_0(0,0)$ is unstable.
• Under some conditions, the coexistence state of system (5) is stable and the alteration in the fear factor’s value has no bearing on this stability.
• We examine the Hopf bifurcation at the unique positive equilibrium. When the fear factor’s value decreases, the limit cycle appears when the fear factor’s value is less than ${\kappa }_H$, and it disappears when the fear factor’s value is equal to ${\kappa }_{het}$ through a loop of heteroclinic orbits.

2. Boundedness and Positivity

Lemma 1. 

For system (5), the first quadrant ${\mathbb{R}}_{+}^2$ is a positive invariant set.

Proof of Lemma 1. 

For system (5), it is not difficult to show that the set $\left\{\right(z,s),s=0\}$ is an invariant set. This means that any orbit of system (5) that touches the z-axis stays forever on it. On the other hand, since $z^{\alpha }$ is a nondifferentiable function over $z=0$, the solution of system (5) that belongs in $\left\{\right(z,s),z=0\}$ is not unique. We can reduce system (5) to

$$\begin{array}{cc}\hfill \frac{dz}{dt}& =0,\hfill \\ \hfill \frac{ds}{dt}& =-\delta s.\hfill \end{array}$$

(6)

Along the s-axes, the solution of system (6) moves closer to the origin. This means that any orbit of system (5) that touches the s-axis stays forever on it. □

Lemma 2. 

All solutions of system (5) with an initial value in ${\mathbb{R}}_{+}^2$ are bounded.

Proof of Lemma 2. 

Let $\left(z\right(t),s(t\left)\right)$ be any solution of system (5) with $(z_0,s_0)\in {\mathbb{R}}_{+}^2$. If ${\displaystyle z_0>\frac{\varrho }{\xi }}$, ${\displaystyle \frac{dz}{dt}=\frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2-\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}<\varrho z(1-\frac{z}{\frac{\varrho }{\xi }})<0}$ if ${\displaystyle z>\frac{\varrho }{\xi }}$, this means that $z\left(t\right)$ decreases when ${\displaystyle z>\frac{\varrho }{\xi }}$. When ${\displaystyle z=\frac{\varrho }{\xi }}$, ${\displaystyle \frac{dz}{dt}<\varrho z(1-\frac{z}{\frac{\varrho }{\xi }})-\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}=-\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}<0.}$ Therefore, ${\displaystyle z\left(t\right)<{\lambda }_1=max\{z_0,\frac{\varrho }{\xi }\}}$. Let $x=\beta z+s$, hence

$${\displaystyle \frac{dx}{dt}=\beta (\frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2)-\delta s<\beta (\varrho +\delta )z-\delta x.}$$

Then,

$$\frac{dx}{dt}+\delta x<{\lambda }_2,where{\lambda }_2=\beta (\varrho +\delta ){\lambda }_1>0.$$

According to Lemma (1.1, [50]), we obtain

$$x<\frac{{\lambda }_2}{\delta },t\ge t_0.$$

Then, we have

$$\beta z+s<\frac{\beta \varrho +\delta }{\delta }{\lambda }_1,t\ge t_0.$$

In other words, $s\left(t\right)$ is bounded. □

Remark 1. 

The region ${\displaystyle \{(z,s):z>\frac{\varrho }{\xi },s\ge 0\}}$ has no equilibrium.

3. Non-Persistence

Theorem 1. 

For the initial value $(z_0,s_0)$ in ${\mathbb{R}}_{+}^2$, if

$$z^{1-\alpha }\left(0\right)<\frac{(1-\alpha )\gamma {(1-\mu )}^{\alpha }s\left(0\right)}{((1-\alpha )\varrho +1)(1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha })}$$

(7)

where ${\displaystyle {\lambda }_1=max\{z_0,\frac{\varrho }{\xi }\}}$, then the prey population falls into decay.

Proof of Theorem 1. 

We can easly show that ${\displaystyle s\left(t\right)\ge s_0{e}^{-t},\forall t.}$ From the proof of Lemma 2, recalling ${\displaystyle z\left(t\right)\le {\lambda }_1=max\{z_0,\frac{\varrho }{\xi }\}}$, from the first equation of system (5),

$${\displaystyle \frac{dz}{dt}\le \varrho z-\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}\le \varrho z-\frac{\gamma {(1-\mu )}^{\alpha }{z}^{\alpha }{s}_0{e}^{-t}}{1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha }}.}$$

Suppose that

$${\displaystyle \frac{d\widehat{z}}{dt}=\varrho \widehat{z}-\frac{\gamma {(1-\mu )}^{\alpha }{\widehat{z}}^{\alpha }{s}_0{e}^{-t}}{1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha }}}$$

(8)

with $\widehat{z}\left(0\right)=z\left(0\right)=z_0.$ By using the comparison theorem of ODE, we obtain $z\left(t\right)\le \widehat{z}\left(t\right),\forall t$. To solve Equation (8), suppose $\widehat{z}\left(t\right)=x\left(t\right)e^{\varrho t}$; then,

$${\displaystyle \frac{dx}{dt}=-\frac{s_0\gamma {(1-\mu )}^{\alpha }{x}^{\alpha }{e}^{-\left(\right(1-\alpha )\varrho +1)t}}{1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha }}.}$$

(9)

By direct calculation, we have

$$x^{1-\alpha }\left(t\right)=x^{1-\alpha }\left(0\right)-\frac{(1-\alpha )\gamma {(1-\mu )}^{\alpha }{s}_0}{((1-\alpha )\varrho +1)(1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha })}[1-e^{-\left(\right(1-\alpha )\varrho +1)t}].$$

(10)

By the definition of $x\left(t\right)$, it is clear that $x^{1-\alpha }\left(0\right)\ge 0$ and $x\left(t\right)$ is a decreasing function. Thus, $x\left(\tau \right)=0$ for a certain $\tau$ if and only if

$${\widehat{z}}^{1-\alpha }\left(0\right)=x^{1-\alpha }\left(0\right)<\frac{(1-\alpha )\gamma {(1-\mu )}^{\alpha }{s}_0}{((1-\alpha )\varrho +1)(1+\sigma \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha })}.$$

(11)

It is no secret that $x\left(\tau \right)=0$ means $\widehat{z}\left(\tau \right)=0$. Recalling $z\left(t\right)\le \widehat{z}\left(t\right),\forall t.$ Hence, $z\left(\tau \right)\le 0$ when $\widehat{z}\left(\tau \right)=0,$ since $R_{+}^2$ is an invariant set; then, $\widehat{z}\left(t\right)=0,t\ge \tau .$ □

Theorem 2. 

If $\delta >\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }$, then the predator population falls into decay.

Proof of Theorem 2. 

For any solution $\left(z\right(t),s(t\left)\right)$ of system (5), it is easy to prove that there is $\tau \ge 0$ such that ${\displaystyle z\left(t\right)\le \frac{\varrho }{\xi }}$ for $t\ge \tau$. From the second equation of system (5),

$$\begin{array}{cc}\hfill \frac{ds}{dt}& \le \beta \gamma {(1-\mu )}^{\alpha }{z}^{\alpha }s-\delta s\hfill \\ \hfill & \le (\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }-\delta )s,t\ge \tau .\hfill \end{array}$$

(12)

This implies ${\displaystyle s\left(t\right)\le s_0{e}^{-(\delta -\beta \gamma {(1-\mu )}^{\alpha }{\lambda }_1^{\alpha })},t\ge \tau .}$ Therefore, for $\delta >\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }$, we have ${\displaystyle \underset{t\to \infty }{lim}s\left(t\right)=0.}$ □

4. Steady States and Their Stability

From system (5), z-zero-growth isocline is determined by

$${\displaystyle \frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2=\frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(1-\mu )}^{\alpha }{z}^{\alpha }},}$$

and s-zero-growth isoclines are $s=0$ and ${\displaystyle \frac{\beta \gamma {(z-\mu z)}^{\alpha }}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}=\delta }$. We know that the intersection of z-zero-growth isocline and s-zero-growth isocline yields the equilibrium points. For any equilibrium point $E_{\ast }(z_{\ast },s_{\ast })$, the Jacobian matrix of the system (5) around $E_{\ast }$ is given by

$$J\left(E_{\ast }\right)=\left[\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right]$$

(13)

where

$$\begin{array}{cc}\hfill a_{11}& {\displaystyle =-\zeta +\frac{\alpha {\gamma }^2\sigma s_{\ast }{(1-\mu )}^{2\alpha }{z}_{\ast }^{2\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1\right)}^2}-\frac{\alpha \gamma s_{\ast }{(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1}-2\xi z_{\ast }+\frac{\varrho }{\kappa s_{\ast }+1}}\hfill \\ \hfill a_{12}& =-\frac{\gamma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1}-\frac{\kappa \varrho z_{\ast }}{{(\kappa s_{\ast }+1)}^2}\hfill \\ \hfill a_{21}& =\frac{\alpha \beta \gamma s_{\ast }{(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1}-\frac{\alpha \beta {\gamma }^2\sigma s_{\ast }{(1-\mu )}^{2\alpha }{z}_{\ast }^{2\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1\right)}^2}\hfill \\ \hfill a_{22}& =\frac{\beta \gamma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_{\ast }^{\alpha }+1}-\delta \hfill \end{array}$$

4.1. The Trivial Steady State

The trivial steady state $E_0(0,0)$ always exists. In this equilibrium point, both populations fall into decay. Because of the $z^{\alpha }$ term, system (5) is not linearizable and the Jacobian matrix becomes indeterminate. In other words, (13) cannot be calculated for $z=0$ and $s=0$ to determine the stability of origin. In the next theorems, we will discuss the stability of $E_0$.

Theorem 3. 

If $\varrho <\zeta$, then $E_0$ is stable.

Proof of Theorem 3. 

Let $(z_{\ast }\left(t\right),s_{\ast }\left(t\right))$ be any solution of system (5). From the first equation of system (5),

$$\begin{array}{cc}\hfill \frac{d{z}_{\ast }}{dt}& {\displaystyle \le (\varrho -\zeta )z_{\ast }.}\hfill \end{array}$$

Since $\varrho <\zeta$, ${\displaystyle \underset{t\to \infty }{lim}{z}_{\ast }\left(t\right)=0}$. This means that the prey population falls into decay, and we can reduce system (5) to

$$\begin{array}{cc}\hfill \frac{d{z}_{\ast }}{dt}& =0,\hfill \\ \hfill \frac{d{s}_{\ast }}{dt}& =-\delta s_{\ast }.\hfill \end{array}$$

(14)

It is clear that ${\displaystyle \underset{t\to \infty }{lim}{s}_{\ast }\left(t\right)=0}$. The proof is completed. □

Theorem 4. 

If $\varrho >\zeta$, then $E_0$ is an unstable point.

Proof of Theorem 4. 

Let $\mathsf{\Omega }=\{(z,0)\in {\mathbb{R}}_{+}^2:0<z<\frac{\varrho -\zeta }{\xi }\}$, then ${\displaystyle \underset{t\to \infty }{lim}{z}_{\ast }\left(t\right)=\frac{\varrho -\zeta }{\xi }}$ for any solution $(z_{\ast },0)$ with initial values in $\mathsf{\Omega }$. For $\lambda >0$, let ${\mathsf{\Omega }}_{\lambda }=\{(z,s)\in {\mathbb{R}}_{+}^2:\left|(z,s)\right|<\lambda \}$. It is clear that $\mathsf{\Omega }\cap {\mathsf{\Omega }}_{\lambda }\ne \varphi ,$ for all $\lambda >0.$ Let $0<ϵ<\frac{0.1(\varrho -\zeta )}{\xi }$. Then, for all $\lambda >0$, there is $(z_{\lambda },0)\in \mathsf{\Omega }\cap {\mathsf{\Omega }}_{\lambda }$ such that ${\displaystyle \underset{t\to \infty }{lim}{z}_{\ast }\left(t\right)=\frac{\varrho -\zeta }{\xi }>ϵ.}$ Here, $(z_{\ast }\left(t\right),0)$ is the solution of system (5) with initial value $(z_{\lambda },0)$. Therefore, by the definition of stability, $E_0$ is unstable. □

Example 1. 

For $\kappa =0.25855$, $\zeta =0.57511$, $\xi =0.11722$, $\gamma =0.31937$, $\mu =0.29078$, $\alpha =0.48276$, $\sigma =0.4976$, $\beta =0.19344$, and $\delta =0.34488$.

1. 

Take $\varrho =0.44741$, then $\varrho <\zeta$ and $E_0$ is stable (see Figure 3a);

2. 

Take $\varrho =0.64741$, then $\varrho >\zeta$ and $E_0$ is unstable (see Figure 3b).

Figure 3. Phase plane analysis of system (5). (a) In Example 1, $E_0$ is a stable point when $\varrho <\zeta$. (b) In Example 1, $E_0$ is an unstable point when $\varrho >\zeta$.

4.2. The Predator-Free Steady State

If $\varrho >\zeta$ the predator-free equilibrium point ${\displaystyle E_1(\frac{\varrho -\zeta }{\xi },0)}$ exists, which means that the predator becomes extinct and the prey survives. At this equilibrium ${\displaystyle E_1(\frac{\varrho -\zeta }{\xi },0)}$, the Jacobian matrix $J_{E_1}$ is given by

$$J_{E_1}=\left[\begin{array}{cc}{b}_{11}& b_{12}\\ 0& b_{22}\end{array}\right],$$

(15)

where $b_{11},b_{12},$ and $b_{22}$ are given by

$$\begin{array}{cc}\hfill b_{11}& =-2(\varrho -\zeta )-\zeta +\varrho \hfill \\ \hfill b_{12}& =-\frac{\gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }+1}-\frac{\kappa \varrho (\varrho -\zeta )}{\xi }\hfill \\ \hfill b_{22}& =\frac{\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }+1}-\delta \hfill \end{array}$$

Next, we present a theorem on the stability of $E_1$.

Theorem 5. 

(i)

Assuming that $\varrho >\zeta$ and ${\theta }_1<{\theta }_2$ hold, then ${\displaystyle E_1(\frac{\varrho -\zeta }{\xi },0)}$ is a stable node.

(ii)

Assuming that $\varrho >\zeta$ and ${\theta }_1>{\theta }_2$ hold, then ${\displaystyle E_1(\frac{\varrho -\zeta }{\xi },0)}$ is an unstable saddle point.

Here,

$$\begin{array}{cc}\hfill {\theta }_1& =\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }\hfill \\ \hfill {\theta }_2& =\gamma \delta \sigma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }+\delta \hfill \end{array}$$

Proof of Theorem 5. 

The eigenvalues of $J_{E_1}$ are ${\lambda }_1=\zeta -\varrho <0$ and

$${\lambda }_2=\frac{\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }-\gamma \delta \sigma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }-\delta }{\gamma \sigma {(1-\mu )}^{\alpha }{\left(\frac{\varrho -\zeta }{\xi }\right)}^{\alpha }+1}.$$

If ${\theta }_1<{\theta }_2$, then ${\lambda }_2<0,$ and hence $E_1$ is a stable node. If ${\theta }_1>{\theta }_2$, then ${\lambda }_2>0,$ and hence $E_1$ is an unstable saddle point. □

Example 2. 

For $\varrho =0.58201$, $\kappa =0.30764$, $\zeta =0.11826$, $\xi =0.25073$, $\gamma =0.44606$, $\mu =0.14733$, $\alpha =0.26169$, $\sigma =0.39219$, $\beta =0.21466$, and $\delta =0.31582$.

• $\varrho >\zeta$, system (2) has a non-trivial boundary equilibrium $E_1(1.8496,0)$;
• ${\theta }_1=0.1079<0.3238={\theta }_2$ and $E_1$ is a stable node (see Figure 4a).

Figure 4. Phase plane analysis of system (5). (a) In Example 2, $E_1$ exists and it is a stable node when $\varrho >\zeta$ and ${\theta }_1<{\theta }_2$. (b) In Example 3, $E_1$ exists and it is an unstable saddle point when $\varrho >\zeta$ and ${\theta }_1>{\theta }_2$.

Example 3. 

For $\varrho =0.48636$, $\kappa =0.38314$, $\zeta =0.12118$, $\xi =0.18622$, $\gamma =0.58601$, $\mu =0.53635$, $\alpha =0.39764$, $\sigma =0.34461$, $\beta =,0.59255$, and $\delta =0.25618$.

• $\varrho >\zeta$, system (3) has a non-trivial boundary equilibrium $E_1(1.9610,0)$;
• ${\theta }_1=0.3343>0.3060={\theta }_2$ and $E_1$ is an unstable saddle point (see Figure 4b).

4.3. The Steady State of Coexistence

Theorem 6. 

• If $\beta <\sigma \delta$, then system (5) has no positive equilibrium.
• If $\varrho <\zeta +\xi z_1$, then system (5) has no positive equilibrium.
• If $\beta >\sigma \delta$ and $\varrho >\zeta +\xi z_1$, then system (5) has a unique positive equilibrium $E_2(z_1,s_1)$. Moreover, $s_1$ is strictly decreasing with respect to κ

where

$$\begin{array}{cc}\hfill z_1& {\displaystyle =z={\left(\frac{\delta }{\gamma {(1-\mu )}^{\alpha }(\beta -\sigma \delta )}\right)}^{\frac{1}{\alpha }},}\hfill \\ \hfill s_1& =\frac{-({\vartheta }_1\kappa +{\vartheta }_2)+\sqrt{{({\vartheta }_1\kappa +{\vartheta }_2)}^2+4\kappa {\vartheta }_2(\varrho z_1-{\vartheta }_1)}}{2\kappa {\vartheta }_2},\hfill \\ \hfill {\vartheta }_1& =\zeta z_1+\xi z_1^2,\hfill \\ \hfill {\vartheta }_2& =\frac{\delta }{\beta }.\hfill \end{array}$$

Proof of Theorem 6. 

From the s-zero-growth isocline, ${\displaystyle \frac{\beta \gamma {(z-\mu z)}^{\alpha }}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}-\delta =0}$, therefore

$${\displaystyle z={\left(\frac{\delta }{\gamma {(1-\mu )}^{\alpha }(\beta -\sigma \delta )}\right)}^{\frac{1}{\alpha }}.}$$

It is clear that if $\beta <\sigma \delta$, then $z\notin {\mathbb{R}}^{++}$, which means that there is no positive equilibrium. Now, suppose $\beta >\sigma \delta$, then ${\displaystyle z_1={\left(\frac{\delta }{\gamma {(1-\mu )}^{\alpha }(\beta -\sigma \delta )}\right)}^{\frac{1}{\alpha }}>0.}$ It is not difficult to show that ${\displaystyle \frac{\gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{1+\sigma \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}=\frac{\delta }{\beta }.}$ From the z-zero-growth isocline, ${\displaystyle \frac{\varrho z_1}{1+\kappa s}-{\vartheta }_1-{\vartheta }_{2}s=0}$, thus

$${\vartheta }_2\kappa s^2+({\vartheta }_1\kappa +{\vartheta }_2)s+({\vartheta }_1-\varrho z_1)=0.$$

(16)

It is clear that if $({\vartheta }_1-\varrho z_1)>0$$(\varrho <\zeta +\xi z_1)$, there is no positive root of (16). If $({\vartheta }_1-\varrho z_1)<0$$(\varrho >\zeta +\xi z_1)$, then there exists a unique positive root $s_1$ of (16), where

$$s_1=\frac{-({\vartheta }_1\kappa +{\vartheta }_2)+\sqrt{{({\vartheta }_1\kappa +{\vartheta }_2)}^2+4\kappa {\vartheta }_2(\varrho z_1-{\vartheta }_1)}}{2\kappa {\vartheta }_2}$$

Therefore, there is a unique positive equilibrium $E_2(z_1,s_1)$ (see Figure 5). □

Figure 5. The intersection of z-zero-growth isocline and s-zero-growth isocline yields the equilibrium points. When $z_1<\frac{\varrho -\zeta }{\xi }$, there is a unique positive equilibrium.

The Jacobian matrix of the system (5) around $E_2$ is given by

$$J_{E_2}=\left[\begin{array}{cc}{c}_{11}& c_{12}\\ c_{21}& 0\end{array}\right],$$

(17)

where $c_{11},c_{12},$ and $c_{21}$ are given by

$$\begin{array}{cc}\hfill c_{11}& =-\zeta +\frac{\varrho }{\kappa s_1+1}+\frac{\alpha {\gamma }^2\sigma s_1{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right){}^2}-\frac{\alpha \gamma s_1{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-2\xi z_1\hfill \\ \hfill c_{12}& =-\frac{\kappa \varrho z_1}{\left(\kappa s_1+1\right){}^2}-\frac{\gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}\hfill \\ \hfill c_{21}& =\frac{\alpha \beta \gamma s_1{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-\frac{\alpha \beta {\gamma }^2\sigma s_1{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right){}^2}\hfill \end{array}$$

(18)

and therefore

$$\begin{array}{cc}\hfill tr{J}_{E_2}& =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1+\frac{\alpha {\gamma }^2\sigma s_1{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{\left(1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }\right){}^2}-\frac{\alpha \gamma s_1{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}\hfill \\ \hfill det{J}_{E_2}& =(\frac{\kappa \varrho z_1}{\left(\kappa s_1+1\right){}^2}+\frac{\gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1})(\frac{\alpha \beta \gamma s_1{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-\frac{\alpha \beta {\gamma }^2\sigma s_1{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right){}^2})\hfill \end{array}$$

(19)

since ${\displaystyle \frac{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}<1}$, thus $det{J}_{E_2}>0.$

5. The Effect of Fear

In this part, we will examine the effect of fear on the dynamics of system (5) by performing bifurcation analysis, taking the level of fear $\kappa$ as a bifurcation parameter. Let us look at the $tr{J}_{E_2}$ sign. Recalling that the s-zero-growth isocline is ${\displaystyle \frac{\gamma {(z-\mu z)}^{\alpha }s}{1+\sigma \gamma {(z-\mu z)}^{\alpha }}=\frac{\delta }{\beta }s}$, therefore, by substituting it into the z-zero-growth isocline, we obtain ${\displaystyle s=\frac{\beta }{\delta }\left(\frac{\varrho z}{1+\kappa s}-\zeta z-\xi z^2\right)}$. Hence,

$$\begin{array}{cc}\hfill tr{J}_{E_2}& =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1+\frac{\alpha {\gamma }^2\sigma s_1{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{\left(1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }\right){}^2}-\frac{\alpha \gamma s_1{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}\hfill \\ \hfill & =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1+\left[\frac{\alpha \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}\right]\left[\frac{\sigma \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}-1\right]s_1\hfill \\ \hfill & {\displaystyle =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1+\alpha z_1^{-1}\left[\frac{\gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{1+\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}\right]\left(\frac{\sigma \delta -\beta }{\beta }\right)\frac{\beta }{\delta }\left(\frac{\varrho z_1}{1+\kappa s}-\zeta z_1-\xi z_1^2\right)}\hfill \\ \hfill & {\displaystyle =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1+\alpha z_1^{-1}\frac{\delta }{\beta }\left(\frac{\sigma \delta -\beta }{\beta }\right)\frac{\beta }{\delta }\left(\frac{\varrho z_1}{1+\kappa s}-\zeta z_1-\xi z_1^2\right)}\hfill \\ \hfill & {\displaystyle =\frac{\varrho }{\kappa s_1+1}-\zeta -2\xi z_1-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\frac{\varrho }{1+\kappa s}-\zeta -\xi z_1\right)}\hfill \\ \hfill & {\displaystyle =\left(\frac{\varrho }{\kappa s_1+1}\right)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\frac{\varrho }{\kappa s_1+1}\right)-\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1.}\hfill \end{array}$$

Define ${\displaystyle s^{\ast }\left(\kappa \right)=\kappa s_1=\frac{-({\vartheta }_1\kappa +{\vartheta }_2)+\sqrt{{({\vartheta }_1\kappa +{\vartheta }_2)}^2+4\kappa {\vartheta }_2(\varrho z_1-{\vartheta }_1)}}{2{\vartheta }_2}}$. It is clear that $s^{\ast }$ increases with respect to $\kappa$, ${\displaystyle \underset{\kappa \to 0}{lim}{s}^{\ast }\left(\kappa \right)=0}$, and ${\displaystyle \underset{\kappa \to \infty }{lim}{s}^{\ast }\left(\kappa \right)=:s_{\infty }^{\ast }=\frac{\varrho }{\zeta +\xi z_1}-1>0}$. Define ${tr}^{\ast }{J}_{E_2}$:$(0,s_{\infty }^{\ast })$${\displaystyle \mapsto }$$\mathbb{R}$ as follows ${\displaystyle {tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)=\left(\frac{\varrho }{s^{\ast }+1}\right)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\frac{\varrho }{s^{\ast }+1}\right)-\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1}$. It is clear that ${tr}^{\ast }{J}_{E_2}$ decreases with respect to $s^{\ast }$, ${\displaystyle \underset{s^{\ast }\to 0}{lim}{tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)=\varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1}$, and ${\displaystyle \underset{s^{\ast }\to s_{\infty }^{\ast }}{lim}{tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)=-\xi z_1<0}$. If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1<0}$, then ${tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)<0$ for any $s^{\ast }$, which means $tr{J}_{E_2}<0$ for any $\kappa .$ If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1>0}$, then there is a unique $s_H^{\ast }\in (0,s_{\infty }^{\ast })$ such that ${tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)>0$ on $(0,s_H^{\ast })$, and ${tr}^{\ast }{J}_{E_2}\left(s^{\ast }\right)<0$ on $(s_H^{\ast },s_{\infty }^{\ast })$, which means there is a unique ${\kappa }_H\in (0,\infty )$ such that $tr{J}_{E_2}\left(\kappa \right)>0$ on $(0,{\kappa }_H)$, and $tr{J}_{E_2}\left(\kappa \right)<0$ on $({\kappa }_H,\infty )$, where ${\kappa }_H$ satisfies the following equation:

$${\displaystyle \frac{-({\vartheta }_1{\kappa }_H+{\vartheta }_2)+\sqrt{{({\vartheta }_1{\kappa }_H+{\vartheta }_2)}^2+4{\kappa }_H{\vartheta }_2(\varrho z_1-{\vartheta }_1)}}{2{\vartheta }_2}=-\frac{A+\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\varrho \right)-\varrho }{A}}$$

(20)

where $A={\displaystyle (1-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right))\left(\zeta +\xi z_1\right)+\xi z_1}$.

Theorem 7. 

Suppose that $\beta >\sigma \delta$ and $\varrho >\zeta +\xi z_1.$

1. 

If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1<0}$, the unique equilibrium $E_2$ is locally asymptotically stable.

2. 

If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1>0}$, there exists a unique ${\kappa }_H\in (0,\infty )$ such that $E_2$ is unstable when $\kappa <{\kappa }_H$, and locally asymptotically stable when $\kappa >{\kappa }_H$, where ${\kappa }_H$ are defined in Equation (20). In addition, system (5) undergoes a Hopf bifurcation at $E_2$ when $\kappa ={\kappa }_H$, where ${\kappa }_H$ is defined in (20).

Proof of Theorem 7. 

From Theorem 6, there is a unique positive equilibrium $E_2$ if $\beta >\sigma \delta$ and $\varrho >\zeta +\xi z_1$. Recall that $det{J}_{E_2}>0.$ Hence, $E_2$ may be either a focus or node, and its stability is determined by the sign of $tr{J}_{E_2}$. If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1<0}$, then $tr{J}_{E_2}<0$ for any $\kappa$, and $E_2$ is always locally asymptotically stable. If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1>0}$, then $E_2$ is unstable when $\kappa <{\kappa }_H$, and locally asymptotically stable when $\kappa >{\kappa }_H$. Furthermore, $tr{J}_{E_2}=0$ when $\kappa ={\kappa }_H$, and the eigenvalues of ${\displaystyle J_{E_2}}$ are $r=\pm i\sqrt{det{J}_{E_2}}$. Let $r=o\left(\kappa \right)\pm i\lambda \left(\kappa \right)$ be the roots of $r^2-tr{J}_{E_2}r+det{J}_{E_2}=0$ when $\kappa$ near ${\kappa }_H$, then $o\left(\kappa \right)=\frac{tr{J}_{E_2}}{2}$. We have

$$o^{\prime }\left(\kappa \right)=\frac{1}{2}\left(1-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\right)\left(\frac{-\varrho s^{{\ast }^{\prime }}\left(\kappa \right)}{{(s^{\ast }\left(\kappa \right)+1)}^2}\right).$$

(21)

Since $s^{{\ast }^{\prime }}\left({\kappa }_H\right)>0$, $o^{\prime }\left(\kappa \right)\ne 0$, as a result, the transversality condition is satisfied and system (5) undergoes a Hopf bifurcation at $E_2$ when $\kappa ={\kappa }_H$. □

We must compute the normal form close to the Hopf bifurcation point using $\kappa$ as the bifurcation parameter in order to ascertain the properties of the bifurcation. The following truncated normal form has been calculated by Du et al. in [43] using the steps in [51].

$$\begin{array}{cc}\hfill {\rho }^{\prime }& =(\kappa -{\kappa }_H)o^{\prime }\left({\kappa }_H\right)\rho +a\left({\kappa }_H\right){\rho }^3+O({\left|\kappa -{\kappa }_H\right|}^2\rho ,\left|\kappa -{\kappa }_H\right|{\rho }^3,{\rho }^5),\hfill \\ \hfill {\phi }^{\prime }& =\lambda \left({\kappa }_H\right)+(\kappa -{\kappa }_H){\lambda }^{\prime }\left({\kappa }_H\right)+b\left({\kappa }_H\right){\rho }^2+O({\left|\kappa -{\kappa }_H\right|}^2,\left|\kappa -{\kappa }_H\right|{\rho }^2,{\rho }^4).\hfill \end{array}$$

(22)

Recalling that $o^{\prime }\left(\kappa \right)<0$, the properties of Hopf bifurcation are determined by a(${\kappa }_H$), which can be computed by (25) in Section 6.

Theorem 8. 

If $\beta >\sigma \delta$, $\varrho >\zeta +\xi z_1,$ and ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1>0}$, system (5) undergoes a Hopf bifurcation at $E_2$ when $\kappa ={\kappa }_H$.

1. 

If a(${\kappa }_H$) > 0, the bifurcation periodic solution is unstable, and it is bifurcating from $E_2$ as κ increases and passes ${\kappa }_H$.

2. 

If a(${\kappa }_H$) < 0, the bifurcation periodic solution is orbitally asymptotically stable, and it is bifurcating from $E_2$ as κ decreases and passes ${\kappa }_H$.

6. Direction of Hpof Bifurcation with $\kappa$ as Bifurcation Parameter

When $\kappa ={\kappa }_H$, we have $tr{J}_{E_2}=0$, and $\pm i\lambda \left({\kappa }_H\right)=\pm i\sqrt{det{J}_{E_2}}$ are the eigenvalues of the Jacobian matrix at $(z_1,s_H)$. Let $\widehat{z}=z-z_1$ and $\widehat{s}=s-s_H$, and system (5) becomes

$$\begin{array}{cc}\hfill \frac{d\widehat{z}}{dt}& =b_{11}\widehat{z}+b_{12}\widehat{s}+X_1(\widehat{z},\widehat{s}),\hfill \\ \hfill \frac{d\widehat{s}}{dt}& =b_{21}\widehat{z}+b_{22}\widehat{s}+X_2(\widehat{z},\widehat{s}).\hfill \end{array}$$

(23)

where

$$\begin{array}{cc}\hfill X_1(\widehat{z},\widehat{s})& =\frac{1}{2}{u}_{20}{\widehat{z}}^2+u_{11}\widehat{z}\widehat{s}+\frac{1}{2}{u}_{02}{\widehat{s}}^2+\frac{1}{6}{u}_{30}{\widehat{z}}^3+\frac{1}{2}{u}_{21}{\widehat{z}}^2\widehat{s}+\frac{1}{2}{u}_{12}\widehat{z}{\widehat{s}}^2+\frac{1}{6}{u}_{03}{\widehat{s}}^3,\hfill \\ \hfill X_2(\widehat{z},\widehat{s})& =\frac{1}{2}{v}_{20}{\widehat{z}}^2+v_{11}\widehat{z}\widehat{s}+\frac{1}{2}{v}_{02}{\widehat{s}}^2+\frac{1}{6}{v}_{30}{\widehat{z}}^3+\frac{1}{2}{v}_{21}{\widehat{z}}^2\widehat{s}+\frac{1}{2}{v}_{12}\widehat{z}{\widehat{s}}^2+\frac{1}{6}{v}_{03}{\widehat{s}}^3,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill b_{11}=& -\zeta +\frac{\alpha {\gamma }^2\sigma s_H{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2}-\frac{\alpha \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-2\xi z_1+\frac{\varrho }{\kappa s_H+1},\hfill \\ \hfill b_{12}=& -\frac{\gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-\frac{\kappa \varrho z_1}{{(\kappa s_H+1)}^2},\hfill \\ \hfill b_{21}=& \frac{\alpha \beta \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2},\hfill \\ \hfill b_{22}=& \frac{\beta \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha }}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}-\delta ,\hfill \\ \hfill u_{20}=& -2\xi -\frac{2{\alpha }^2{\gamma }^3{\sigma }^2{s}_H{(1-\mu )}^{3\alpha }{z}_1^{3\alpha -2}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^3}+\frac{{\alpha }^2{\gamma }^2\sigma s_H{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -2}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2}-\frac{(\alpha -1)\alpha \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -2}}{\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1}\hfill \\ \hfill & +\frac{\alpha (2\alpha -1){\gamma }^2\sigma s_H{(1-\mu )}^{2\alpha }{z}_1^{2\alpha -2}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2},\hfill \\ \hfill u_{11}=& -\frac{\alpha \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2}-\frac{\kappa \varrho }{{(\kappa s_H+1)}^2},\hfill \\ \hfill u_{02}=& \frac{2{\kappa }^2\varrho z_1}{{(\kappa s_H+1)}^3},\hfill \\ \hfill u_{30}=& -\frac{\alpha \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -3}\left(3\alpha \left({\gamma }^2{\sigma }^2{(1-\mu )}^{2\alpha }{z}_1^{2\alpha }-1\right)+2{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^4}\hfill \\ \hfill & -\frac{\alpha \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -3}\left({\alpha }^2\left({\gamma }^2{\sigma }^2{(1-\mu )}^{2\alpha }{z}_1^{2\alpha }-4\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^4},\hfill \\ \hfill u_{21}=& \frac{\alpha \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha -2}\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+\alpha \left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }-1\right)+1\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^3},\hfill \\ \hfill u_{12}=& \frac{2{\kappa }^2\varrho }{{(\kappa s_H+1)}^3},\hfill \\ \hfill u_{03}=& -\frac{6{\kappa }^3\varrho z_1}{{(\kappa s_H+1)}^4},\hfill \\ \hfill v_{20}=& -\frac{\alpha \beta \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -2}\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+\alpha \left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }-1\right)+1\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^3},\hfill \\ \hfill v_{11}=& \frac{\alpha \beta \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha -1}}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2},\hfill \\ \hfill v_{02}=& 0,\hfill \\ \hfill v_{30}=& \frac{\alpha \beta \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -3}\left(3\alpha \left({\gamma }^2{\sigma }^2{(1-\mu )}^{2\alpha }{z}_1^{2\alpha }-1\right)+2{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^2\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^4}\hfill \\ \hfill & +\frac{\alpha \beta \gamma s_H{(1-\mu )}^{\alpha }{z}_1^{\alpha -3}\left({\alpha }^2\left({\gamma }^2{\sigma }^2{(1-\mu )}^{2\alpha }{z}_1^{2\alpha }-4\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^4},\hfill \\ \hfill v_{21}=& -\frac{\alpha \beta \gamma {(1-\mu )}^{\alpha }{z}_1^{\alpha -2}\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+\alpha \left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }-1\right)+1\right)}{{\left(\gamma \sigma {(1-\mu )}^{\alpha }{z}_1^{\alpha }+1\right)}^3},\hfill \\ \hfill v_{12}=& 0,\hfill \\ \hfill v_{03}=& 0.\hfill \end{array}$$

Now, let $z=\widehat{z}$, $s=\frac{1}{\lambda \left({\kappa }_H\right)}(b_{11}\widehat{z}+b_{12}\widehat{s})$; then, system (23) becomes

$$\begin{array}{cc}\hfill \frac{dz}{dt}& =-\lambda \left({\kappa }_H\right)s+G_1(z,s),\hfill \\ \hfill \frac{ds}{dt}& =\lambda \left({\kappa }_H\right)+G_2(z,s).\hfill \end{array}$$

(24)

where

$$\begin{array}{cc}\hfill F(z,s)=& X_1(z,-\frac{b_{11}z+\lambda \left({\kappa }_H\right)s}{b_{12}}),\hfill \\ \hfill G(z,s)=& -\frac{1}{\lambda \left({\kappa }_H\right)}\left(b_{11}{X}_1(z,-\frac{b_{11}z+\lambda \left({\kappa }_H\right)s}{b_{12}}))+b_{12}{X}_2(z,-\frac{b_{11}z+\lambda \left({\kappa }_H\right)s}{b_{12}})\right).\hfill \end{array}$$

From [51], $a\left({\kappa }_H\right)$ in (22) can be obtained by

$$a\left({\kappa }_H\right)=\frac{1}{16}[F_{zzz}+F_{zss}+G_{zzs}+G_{sss}]+\frac{1}{16\lambda \left({\kappa }_H\right)}[F_{zs}(F_{zz}+F_{ss})-G_{zs}(G_{zz}+G_{ss})-F_{zz}{G}_{zz}+F_{ss}{G}_{ss}]$$

(25)

7. Examples and Simulations

Example 4. 

Choose $\varrho =0.81016$, $\zeta =0.50763$, $\xi =0.47125$, $\gamma =0.49284$, $\mu =0.2$, $\alpha =0.461621$, $\sigma =0.30155$, $\beta =0.51521$, and $\delta =0.11491$. Then, $\beta >\sigma \delta$, $z_1=0.2609$, and, $\varrho >\zeta +\xi z_1$. When $\kappa$ changes, there is a unique positive equilibrium $E_2(z_1,s_1)$. Since ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1=-0.0207<0}$, thus, from Theorem 7, $E_2$ is locally asymptotically stable (see Figure 6).

Figure 6. In Example 4, the stability of $E_2$ is unaffected by the variation in the value of k; $E_2$ is locally asymptotically stable. (a) $\kappa =0.01$ (b) $\kappa =0.99$.

Example 5. 

Choose $\varrho =0.91016$, $\zeta =0.50763$, $\xi =0.47125$, $\gamma =0.49284$, $\mu =0.2$, $\alpha =0.461621$, $\sigma =0.30155$, $\beta =0.51521$, and $\delta =0.11491$. Then, $\beta >\sigma \delta$, $z_1=0.2609$, and, $\varrho >\zeta +\xi z_1$. When $\kappa$ changes, there is a unique positive equilibrium $E_2(z_1,s_1)$. Since ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1=0.0362>0}$, thus, from Theorem 7, there exists a unique ${\kappa }_H=0.297765$ such that $E_2$ is unstable when $\kappa <{\kappa }_H$ (see Figure 7), and locally asymptotically stable when $\kappa >{\kappa }_H$ (see Figure 8). In addition, system (5) undergoes a Hopf bifurcation at $E_2$ when $\kappa ={\kappa }_H$ (see Figure 9a). Actually, by using the procedures in Section 6, we can determine $a\left({\kappa }_H\right)=-0.134214<0$; therefore, from Theorem 8, the bifurcation periodic solution is orbitally asymptotically stable (see Figure 9b), and it is bifurcating from $E_2$ as $\kappa$ decreases and passes ${\kappa }_H$. There is a unique limit cycle that appears when $\kappa <{\kappa }_H$ (see Figure 9c–e) and, through a loop of heteroclinic orbits, the limit cycle vanishes when $\kappa$ decreases to ${\kappa }_{het}=0.14714$ (see Figure 9f). The figures were drawn by Wolfram Mathematica [52].

Figure 7. Choose $\varrho =0.91016$, $\zeta =0.50763$, $\xi =0.47125$, $\gamma =0.49284$, $\mu =0.2$, $\alpha =0.461621$, $\sigma =0.30155$, $\beta =0.51521$, and $\delta =0.11491$. $E_2$ is unstable when $\kappa =0.125<{\kappa }_H$.

Figure 8. Choose $\varrho =0.91016$, $\zeta =0.50763$, $\xi =0.47125$, $\gamma =0.49284$, $\mu =0.2$, $\alpha =0.461621$, $\sigma =0.30155$, $\beta =0.51521$, and $\delta =0.11491$. $E_2$ is locally asymptotically stable when $\kappa =0.4226>{\kappa }_H$.

Figure 9. Choose $\varrho =0.91016$, $\zeta =0.50763$, $\xi =0.47125$, $\gamma =0.49284$, $\mu =0.2$, $\alpha =0.461621$, $\sigma =0.30155$, $\beta =0.51521$, and $\delta =0.11491$. (a) Bifurcation diagram of system (5) in Example 5. (b) The bifurcation periodic solution is orbitally asymptotically stable. (c,d) There is a unique limit cycle that appears when $\kappa =0.28<{\kappa }_H$. (e) Dynamics of system 5 in Example 5 when $\kappa =0.28$. (f) When $\kappa ={\kappa }_{het}=0.14714$, there is a loop of heteroclinic orbits.

8. Discussion

A predator–prey system including group defense in the prey, the fear factor, and the refuge is proposed and investigated in this paper. The main goal of this study is to find the answer to the following question: how do group defense, the fear factor, and the refuge affect the qualitative dynamics of the model? According to the model presented in this paper, the functional response is classified as type IV, and it is nondifferentiable on the s-axis. The functional response ultimately increases (when $z\ge \frac{1}{1-\mu }$) as the efficiency of aggregation for prey increases, and it decreases as the refuge capacity increases. We found the following dynamic behaviors in system (5):

• According to Theorem 2, when $\delta >\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }$, the predator population is non-persistent, i.e., the predator population falls into decay if the per capita death rate of the predator is greater than a constant $\theta =\beta \gamma {(1-\mu )}^{\alpha }{\left(\frac{\varrho }{\xi }\right)}^{\alpha }$ that depends on several parameters. Note that this $\theta$ decreases as the capacity of a refuge at t increases, and $\theta$ increases (decreases) as the value of the efficiency of aggregation for prey increases if $\frac{(1-\mu )\varrho }{\xi }>1$ ($\frac{(1-\mu )\varrho }{\xi }<1$).
• In system (5), there is a maximum of three equilibria, including a positive one. The trivial steady state $E_0$ always exists, the predator-free steady state exists when $\varrho >\zeta$, and system (5) has a unique positive equilibrium when $\beta >\sigma \delta$ and ${\displaystyle \varrho >\zeta +\xi {\left(\frac{\delta }{\beta \gamma {(1-\mu )}^{\alpha }-\sigma \delta \gamma {(1-\mu )}^{\alpha }}\right)}^{\frac{1}{\alpha }}}$.
• Because of the $z^{\alpha }$ term, the Jacobian matrix is indeterminate at the origin. Therefore, it is impossible to carry out a stability analysis by simply looking at its eigenvalues. We used the definition of stability to prove that if $\varrho <\zeta$, then $E_0(0,0)$ is stable, and if $\varrho >\zeta$, then $E_0(0,0)$ is unstable.
• If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1<0}$, the coexistence state of system (5) is stable and the alteration in the fear factor’s value has no bearing on this stability.
• If ${\displaystyle \varrho -\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\varrho -\left[(\zeta +\xi z_1)-\alpha \left(\frac{\beta -\sigma \delta }{\beta }\right)\left(\zeta +\xi z_1\right)\right]-\xi z_1>0}$, we examine the Hopf bifurcation at the unique positive equilibrium. When the fear factor’s value decreases, the limit cycle appears when the fear factor’s value is less than ${\kappa }_H$, and it disappears when the fear factor’s value is equal to ${\kappa }_{het}$ through a loop of heteroclinic orbits.