Dynamics in a Predator–Prey Model with Cooperative Hunting and Allee Effect

Abstract

This paper deals with a diffusive predator–prey model with two delays. First, we consider the local bifurcation and global dynamical behavior of the kinetic system, which is a predator–prey model with cooperative hunting and Allee effect. For the model with weak cooperation, we prove the existence of limit cycle, and a loop of heteroclinic orbits connecting two equilibria at a threshold of conversion rate $p=p^{\#}$, by investigating stable and unstable manifolds of saddles. When $p>p^{\#}$, both species go extinct, and when $p<p^{\#}$, there is a separatrix. The species with initial population above the separatrix finally become extinct, and the species with initial population below it can be coexisting, oscillating sustainably, or surviving of the prey only. In the case with strong cooperation, we exhibit the complex dynamics of system, including limit cycle, loop of heteroclinic orbits among three equilibria, and homoclinic cycle with the aid of theoretical analysis or numerical simulation. There may be three stable states coexisting: extinction state, coexistence or sustained oscillation, and the survival of the prey only, and the attraction basin of each state is obtained in the phase plane. Moreover, we find diffusion may induce Turing instability and Turing–Hopf bifurcation, leaving the system with spatially inhomogeneous distribution of the species, coexistence of two different spatial-temporal oscillations. Finally, we consider Hopf and double Hopf bifurcations of the diffusive system induced by two delays: mature delay of the prey and gestation delay of the predator. Normal form analysis indicates that two spatially homogeneous periodic oscillations may coexist by increasing both delays.

1. Introduction

A predator–prey model can be described by the following equations [1]:

$$\left\{\begin{array}{ccc}\dot{u}\hfill & =\hfill & f\left(u\right)u-G(u,v)v,\hfill \\ \dot{v}\hfill & =\hfill & dG(u,v)v-mv,\hfill \end{array}\right.$$

(1)

where u and v stand for the densities of prey and predator, respectively. $f\left(u\right)$ represents the per capita prey growth rate in the absence of predators, and $G(u,v)$ is the functional response characterizing predation. d describes the rate of biomass conversion from predation, and m is the death rate of predator. Many kinds of functional response have been proposed [2]. Among them, Holling type I, II, and III [3,4,5,6] are discussed widely, which are prey-dependent functional response.

Cooperative behavior within a species is very common in nature, of which cooperative hunting is the most widely distributed form in animals. In mammals, the most famous examples include wolves [7], African wild dogs [8], lions [9], and chimpanzees [10], who cooperate for hunting preys. Cooperative behaviors are also widespread in other living organisms, such as aquatic organisms [11], spiders [12], birds [13], and ants [14], who find, attack, and move their preys together. Due to hunting cooperation, the attack rate of predators increases with predator density, which obviously depends on both prey and predator densities. Recently, a few literatures have paid attention to derive functional response to describe the cooperative hunting [1,15,16]. Cosner et al. [1] proposed a functional response to explore the effects of predator aggregation when predators encounter a cluster of prey. Berec [15] generalized the Holling type-II functional response to a family of functional responses by considering attack rate and handling time of predators varying with predator density to interpret the foraging facilitation among predators. Alves and Hilker [16] considered a special case of the more general functional response proposed by Berec [15]. They added a cooperation term to the attack rate for representing the benefits that hunting cooperation brings to the predator population, which makes the functional response $G(u,v)=(b+cv)u$. Here, $c>0$ describes the strength of predator cooperation in hunting, and $cv$ is referred to as the cooperation term. They found that cooperative hunting can improve persistence of the predator, and it is a form of foraging facilitation which can induce strong Allee effects. After their work, various types of biological factors (such as Allee effect, fear effect, and so on) and spatial diffusion are incorporated into continuous-time predator–prey models with cooperative hunting [17,18,19,20,21,22]. The impact of hunting cooperation in discrete-time predator–prey models have also been investigated [23,24,25,26]. Pal et al. [23] observed that hunting cooperation can modify the period-doubling route to chaos by reverse period-halving bifurcations and makes the system stable. Chow et al. [26] derived a sufficient condition for which both populations can coexist.

The most well-known example of $f\left(u\right)$ is the logistic form $f\left(u\right)=r\left(1-\frac{u}{K}\right)$, where r is the intrinsic growth rate and K denotes the environmental carrying capacity. Then, in the absence of predator, the prey population is governed by $\dot{u}=ru\left(1-\frac{u}{K}\right)$. We can conclude from the equation that the species may increase in size when the density is low. However, for many species, low population density may induce many problems, such as mate difficulties, inbreeding, and predator avoidance of defense. It turns out that the growth rate of the low density population is not always positive, and it may be negative when the density of population is less than the minimum number a for the survival of the population, which is called the Allee threshold. The phenomenon is known as the Allee effect, which was first observed by Allee [27], and has been observed on various organisms, such as vertebrates, invertebrates, and plants. Such a population can be described by $\dot{u}=ru\left(1-\frac{u}{K}\right)(u-a)$. The Allee effect may cause the extinction of a low-density population. Because of increasing fragmentation of habitats and invasions of exotic species, the Allee effect has aroused increasing attention (see, for example, in [28,29,30] and the references cited therein).

Jang et al. [17] considered both the cooperation in the predator and Allee effects in the prey, and proposed the following predator–prey model:

$$\left\{\begin{array}{ccc}\dot{u}\hfill & =\hfill & r_{1}u\left[1-\frac{u}{K_1}\right]\left[u-a_1\right]-\left[b_1+c_{1}v\right]uv,\hfill \\ \dot{v}\hfill & =\hfill & p_1[b_1+c_{1}v]uv-m_{1}v,\hfill \end{array}\right.$$

(2)

where $r_1$ is the per capita intrinsic growth rate of the prey, $K_1$ is the environmental carrying capacity for the prey, $a_1$ ($a_1<K_1$) is the Allee threshold of the prey population, $b_1$ is the attack rate per predator and prey, $c_1$ measures the degree of cooperation of predator, $p_1$ is the prey conversion to predator, and $m_1$ is the per capita death rate of predator. They discussed the existence and stability of the equilibria and the occurrence of Hopf bifurcation. They also devised a best strategy for culling the predator by maximizing the prey population and minimizing the predators along with the costs associated with the control.

In fact, the preys and predators distribute inhomogeneously in different locations, and the spatial diffusion plays an important part in the process of population evolution. Turing instability and Turing–Hopf bifurcation induced by diffusion have been widely investigated recently (see in [31,32,33,34]).

It has been widely accepted that time delays have very complex effect on the dynamics of a system, for example, some delays can destroy the stability of equilibria and induce various oscillations and periodic solutions (see, for example, in [35,36,37,38] and references cited therein). There are time delays in almost every process of population interaction, so it is more realistic to introduce time delays when we model the interaction of predator and prey. For example, after consuming the prey, the corresponding biomass cannot be added to the predator population immediately. Thus, the reproduction of predator after predating the prey will be mediated by some discrete time lag (say ${\mathrm{\tau }}_2$) required for gestation of predator, which can be regarded as gestation delay. Therefore, the growth rate of predator depends on the number of individuals present at time $t-{\mathrm{\tau }}_2$. There is an extensive literature about the studies of the dynamics of predator–prey models with the effect of gestation delay of the predator [35,36,37,38]. There is a time lag (say ${\mathrm{\tau }}_1$) in the growth to maturity of the prey, and such a maturation delay of the prey has also been considered [39,40,41,42,43]. However, to the best of our knowledge, there is very little literatures on delayed predator–prey system with Allee effect [44].

Taking into account the diffusion and delays in system (2), we consider the following diffusive system with two delays:

$$\left\{\begin{array}{c}\frac{\partial u(x,t)}{\partial t}=d_1\Delta u(x,t)+r_{1}u(x,t)[1-\frac{u(x,t-{\mathrm{\tau }}_1)}{K_1}](u(x,t)-a_1)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-(b_1+c_{1}v(x,t))u(x,t)v(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ \frac{\partial v(x,t)}{\partial t}=d_2\Delta v(x,t)+p_1\left[b_1+c_{1}v(x,t-{\mathrm{\tau }}_2)\right]u(x,t-{\mathrm{\tau }}_2)v(x,t)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-mv(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ {\partial }_{v}u(x,t)=0, {\partial }_{v}v(x,t)=0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x=0,l\pi , t>0\hfill \\ u(x,t)={\mathsf{\Phi }}_1(x,t)\ge 0, v(x,t)={\mathsf{\Phi }}_2(x,t)\ge 0,\hspace{1em}\hspace{1em}\hspace{1em}x\in [0,l\pi ], t\in [-\mathrm{\tau },0].\hfill \end{array}\right.$$

(3)

Here, ${\mathrm{\tau }}_1$ is the time delay due to the maturation of the prey, ${\mathrm{\tau }}_2$ is time delay due to gestation of the predator, and we define $\mathrm{\tau }=\max \{{\mathrm{\tau }}_1,{\mathrm{\tau }}_2\}$. $d_1,d_2>0$ are the diffusion coefficients characterizing the rates of the spatial dispersion of the prey and predator population, respectively.

Systems with multiple delays have attracted much attention [45,46,47,48,49,50,51]. Generally, delay may induce Hopf bifurcation, and if Hopf bifurcation curves intersect, double Hopf bifurcation may arise. To figure out the effect of delay on the dynamics of systems, Hopf bifurcation and double Hopf bifurcation induced by delay have been investigated [52,53,54,55,56]. However, in most of literatures we mentioned above, the system is reduced into a system with one delay. In fact, systems with multiple delays conform to reality better than one single delay. Recently, the research on the dynamics and bifurcation analysis of system with two simultaneously varying delays are of great interest to scholars. In [49,57], double Hopf bifurcation induced by two delays is studied by different methods. Through the analysis of double Hopf bifurcation, we can classify the topological structures of various bifurcating solutions. By the classification, the dynamics in the neighborhood of the double Hopf bifurcation point in system can be obtained completely.

For the convenience of discussion and reducing the number of parameters, we nondimensionalize the model by letting

$$\widehat{u}=\frac{1}{K_1}u, \widehat{v}=b_{1}v.$$

Dropping the hats for simplicity of notations, we obtain the simplified dimensionless system

$$\left\{\begin{array}{c}\frac{\partial u(x,t)}{\partial t}=d_1\Delta u(x,t)+ru(x,t)[1-u(x,t-{\mathrm{\tau }}_1)](u(x,t)-a)\hfill \\ \hspace{1em}\hspace{1em} -(1+cv(x,t\left)\right)u(x,t)v(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ \frac{\partial v(x,t)}{\partial t}=d_2\Delta v(x,t)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em} +mv(x,t)\left[pu(x,t-{\mathrm{\tau }}_2)(1+cv(x,t-{\mathrm{\tau }}_2))-1\right], x\in (0,l\pi ), t>0\hfill \\ {\partial }_{v}u(x,t)=0, {\partial }_{v}v(x,t)=0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x=0,l\pi , t>0\hfill \\ u(x,t)={\mathsf{\Phi }}_1(x,t)\ge 0, v(x,t)={\mathsf{\Phi }}_2(x,t)\ge 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} x\in [0,l\pi ], t\in [-\mathrm{\tau },0],\hfill \end{array}\right.$$

(4)

where $r=K_1{r}_1, a=\frac{a_1}{K_1}, c=\frac{c_1}{b_1^2}, \mathrm{and} p=\frac{b_1{p}_1{K}_1}{m}$. Recalling the definition of Allee effect, one always has $0<a<1$. In fact, for fixed $K_1$, $r_1$, $a_1$, $b_1$, and m, we have c and p directly proportional to $c_1$ and $p_1$, respectively. Thus, we refer to c and p as the degree of cooperative hunting of the predator and the conversion rate from the prey to the predator, respectively, in the context. By the explanation above, we make the following assumption always:

$$\left(H\right) r, a, c, m, p \mathrm{a}\mathrm{n}\mathrm{d} c \mathrm{a}\mathrm{r}\mathrm{e} \mathrm{a}\mathrm{l}\mathrm{l} \mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e},\mathrm{a}\mathrm{n}\mathrm{d} 0<a<1.$$

The Allee effect can drive the extinction of low-density species, some of which are preyed by predators that engage in cooperative hunting. This paper investigate the dynamics of system (4) with cooperative hunting and Allee effect from the point of view bifurcation analysis, which can provide useful insights into the intersection of the predator and prey, and help us figure out how to prevent such a prey population from extinction. We find out that the number of constant steady states of (4) is highly dependent on the strength of cooperation in predator c. When $c<\frac{1}{r(1-a)}$, there is one or no positive steady states, while when $c>\frac{1}{r(1-a)}$, there is no, one or two positive steady states. We say the cooperation in the former case is weak, and that in the latter is strong. First of all, we figure out the global dynamical behavior of the kinetic system through the analysis of local bifurcation and the stable and unstable manifolds, taking p as a bifurcation parameter. The existence of some connecting orbits is obtained, thus, a partition of the phase space is given, i.e., the attraction basin of each stability state is obtained. For the model with weak cooperation, we give the condition of the stability of the positive steady state and the occurrence of Hopf bifurcation. Through detailed analysis on the stable and unstable manifolds of saddles, we prove theoretically there is a loop of heteroclinic orbits when $p=p^{\#}$. When $p>p^{\#}$, the extinction steady state $E_0(0,0)$ is globally asymptotically stable. When $p<p^{\#}$, there is a separatrix, which is the stable manifold of $E_a$. The orbits above the separatrix converge to $E_0$, and the orbits below it may converge to $E_1$, $E^{*}$ or a limit cycle. Thus, there are three types of bistability: bistability between two boundary equilibria, between a boundary equilibrium and a interior equilibrium, and between a boundary equilibrium and a limit cycle. For the model with strong cooperation, it turns out there are many complex phenomena, including limit cycle, homoclinic cycle, and a loop of heteroclinic orbits between two or among three equilibria. The model may display tristable dynamics, including extinction state, coexistence or sustained oscillation, and the survival of the prey only.

Second, we discuss the diffusive system (4) with no delays. Taking diffusive coefficients as parameters, we give the condition of Turing instability and Turing–Hopf bifurcation in both weak and strong cooperation cases. We discuss the effect of diffusion on the dynamics, and illustrate complex spatio-temporal dynamical behavior, including the existence of spatially inhomogeneous steady state, coexistence of two spatially inhomogeneous periodic solutions.

Third, taking two time delays as bifurcation parameters, we study the effect of two simultaneous varying delays on the dynamics of system (4). After some detailed analysis on the characteristic equation with three exponential terms, the stability switching curves are obtained, on which there is a pair of purely imaginary eigenvalues. An intersection of the stability switching curves may be a double Hopf bifurcation point. We derive the normal form on the center manifold near the double Hopf bifurcation point, and give the unfoldings near the critical points, which indicate that two spatially homogeneous periodic oscillations may coexist by increasing both delays.

The scheme of this paper is arranged as follows. In Section 2, we investigate the existence of constant steady states for system (4). In Section 3, we analyze the local bifurcation and the global dynamics of the kinetic system (7) with weak cooperation and strong cooperation, respectively. In Section 4, we consider the effect of diffusion on system (4) through investigating Turing instability and Turing–Hopf bifurcation. In Section 5, we consider the diffusive system with two delays, and Hopf bifurcation and double Hopf bifurcation induced by two delays are analyzed on the center manifold via normal form approach. In Section 6, we carry out some numerical simulations to support our theoretical analysis and illustrate complex dynamical phenomenon.

2. The Existence of Constant Steady States

First, we consider the existence of steady states of model (4). As (4) is two-dimensional, it is hard to consider non-constant steady states. Thus, we only discuss the existence of constant steady states.

System (4) has three boundary equilibria: (i) $E_0(0,0)$, which means the extinction of both the species; (ii) $E_a(a,0)$, which is induced by Allee effect; and (iii) $E_1(1,0)$, which means the extinction of the predator and the survival of the prey, achieving its carrying capacity.

Now, we discuss the existence of interior equilibria. Obviously, any interior equilibrium has components $u>0$ and $v>0$ satisfying

$$\begin{array}{cc}\hfill r(1-u)(u-a)& =(1+cv)v,\hfill \\ \hfill pu(1+cv)& =1,\hfill \end{array}$$

(5)

i.e.,

$$\begin{array}{cc}\hfill r(1-u)(u-a)& =\frac{1-pu}{c{p}^2{u}^2},\hfill \\ \hfill v& =\frac{1-pu}{cpu}.\hfill \end{array}$$

(6)

In fact, $r(1-u)(u-a)=(1+cv)v$ is an ellipse, sitting in the first and forth quadrant, and intersecting the horizontal axis at a and 1. $pu(1+cv)=1$ intersects the horizontal axis at $\frac{1}{p}$, and is decreasing and concave in $(0,\frac{1}{p})$. Denote the u-nullcline $r(1-u)(u-a)=(1+cv)v$ as $v=f\left(u\right)$, and the v-nullcline $pu(1+cv)=1$ as $v=g\left(u\right)$. The existence of interior equilibria may have the following different cases.

Proposition 1.

(i) When $p\ge \frac{1}{a}$, system (4) has no interior equilibria.

(ii) 

When $1<p<\frac{1}{a}$, system (4) has a unique positive constant equilibrium $E^{*}(u^{*},v^{*})$ with $a<u^{*}<\frac{1}{p}$.

(iii) 

Suppose that $p<1$. If $c<\frac{1}{r(1-a)}$, system (4) has no interior equilibria. If $c>\frac{1}{r(1-a)}$, there exists a $p_{SN}<1$, such that system (4) has

(a) 

two interior equilibria $E^{*}$ and $E_R^{*}$ when $p_{SN}<p<1$;

(b) 

one interior equilibrium $E_R^{**}$ when $p=p_{SN}$; and

(c) 

no interior equilibria when $p<p_{SN}$.

Proof. 

In fact, when $a<\frac{1}{p}<1$, two nullclines have two intersections: $E^{*}$ in the first quadrant and $E_R^{*}$ in the forth quadrant. Thus, there is a unique interior equilibrium $E^{*}(u^{*},v^{*})$ with $u^{*}$ and $v^{*}$ satisfying (5). When $\frac{1}{p}$ decreases to a, $E^{*}$ collides with $E_a$, and $E_R^{*}$ is still in the forth quadrant. When $\frac{1}{p}<a$, $E^{*}$ moves into the forth quadrant, leaving no interior equilibria.

When $\frac{1}{p}$ increases to 1, v-nullcline and u-nullcline intersect at $E_1(1,0)$, with the slopes of tangents at $E_1(1,0)$ being $-\frac{1}{c}$ and $-r(1-a)$, respectively. If $-\frac{1}{c}<-r(1-a)$, $E^{*}$ collides with $E_1$ when $p=1$, $E_R^{*}$ is in the forth quadrant, and there is no interior equilibria (see Figure 1a). Moreover, when $p<1$, there is no interior equilibria. If $-\frac{1}{c}>-r(1-a)$, $E_R^{*}$ (sitting in the forth quadrant when $1<p<\frac{1}{a}$) collides with $E_1$ when $p=1$, and there is a unique interior equilibrium $E^{*}$ in the first quadrant (see Figure 1b). When $p<1$ and close to 1, $E_R^{*}$ moves into the first quadrant, and there are two interior equilibria $E^{*}$ and $E_R^{*}$. If $p<1$ goes on to decrease, there exists a $p_{SN}$, such that the u-nullcline $v=f\left(u\right)$ and v-nullcline $v=g\left(u\right)$ tangent at $E_R^{**}$, where $E^{*}$ and $E_R^{*}$ collide, and there is a unique interior equilibrium $E_R^{**}$. When $p<p_{SN}$, there is no interior equilibria. □

Remark 1.

In the absence of cooperative hunting within the predator, i.e., $c=0$, system (4) always has a unique interior equilibrium $E_0^{*}=(\frac{1}{p},r(1-\frac{1}{p})(\frac{1}{p}-a))$. The predator and prey coexist if and only if $1<p<\frac{1}{a}$, and the predator is distinct if $p<1$. However, from proposition 1, the predator and prey still coexist when $p_{SN}<p<1$ with strong cooperation, which means that strong cooperation is beneficial for the survival of predator.

Proposition 1 tells us that the strength of cooperation $c=\frac{1}{r(1-a)}$ is a threshold for the system (4). To be specific, if $c<\frac{1}{r(1-a)}$, system (4) has at most one coexisting equilibrium, while if $c>\frac{1}{r(1-a)}$, system (4) may have two coexisting equilibria. Thus, in the following discussion, when $c<\frac{1}{r(1-a)}$, we say the cooperation among predators is weak; when $c>\frac{1}{r(1-a)}$, the cooperation is strong.

3. Bifurcation and Global Dynamics of Kinetic System

Before the discussion of the stability of equilibria of system (4), we should first figure out the stability of the equilibria of the kinetic system

$$\left\{\begin{array}{c}\dot{u}=ru(1-u)(u-a)-(1+cv)uv,\hfill \\ \dot{v}=mv\left[pu\left(1+cv\right)-1\right].\hfill \end{array}\right.$$

(7)

We investigate the local bifurcation and global dynamics when the cooperation is weak and strong, i.e., when $c<\frac{1}{r(1-a)}$ and $c>\frac{1}{r(1-a)}$.

3.1. Bifurcation and Global Dynamics of Kinetic System with Weak Cooperative Hunting

From proposition 1, if $c<\frac{1}{r(1-a)}$, system (4) has a unique interior equilibrium. In this section, taking p as bifurcation parameter, we first investigate the Hopf bifurcation near the unique interior equilibrium. Through studying the stable manifold and unstable manifold of saddles, we prove the existence of loop of heteroclinic orbits, and get the global dynamics of system (7).

3.1.1. Stability of All Equilibria and Hopf Bifurcation at $E^{*}$

The Jacobian matrices of the function on the right-hand of system (7) around $E_0$, $E_a$, and $E_1$ are

$$J_{E_0}=\left(\begin{array}{cc}-ra& 0\\ 0& -m\end{array}\right),$$

$$J_{E_a}=\left(\begin{array}{cc}ra(1-a)& -a\\ 0& m(pa-1)\end{array}\right),$$

$$J_{E_1}=\left(\begin{array}{cc}-r(1-a)& -1\\ 0& m(p-1)\end{array}\right),$$

respectively, from which we can easily get the local stability of the boundary equilibria.

Lemma 1.

For system (7),

 (i)

$E_0(0,0)$ is a stable node;

 (ii)

$E_a(a,0)$ is an unstable node if $p>\frac{1}{a}$, and it is a saddle if $p<\frac{1}{a}$; and

 (iii)

$E_1(1,0)$ is a stable node if $p<1$, and it is a saddle if $p>1$.

For the interior equilibrium $E^{*}(u^{*},v^{*})$, the corresponding Jacobian matrix is

$$J_{E^{*}}=\left(\begin{array}{cc}r{u}^{*}(1+a-2u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ mp{v}^{*}(1+c{v}^{*})& mpc{u}^{*}{v}^{*}\end{array}\right),$$

(8)

and thus

$$\begin{array}{c}\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+mpc{u}^{*}{v}^{*}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}),\hfill \\ \det J_{E^{*}}=mp{u}^{*}{v}^{*}\left[rc{u}^{*}(1+a-2u^{*})+(1+c{v}^{*})(1+2c{v}^{*})\right].\hfill \end{array}$$

In fact, u-nullcline $v=f\left(u\right)$ and v-nullcline $v=g\left(u\right)$ intersect at $E^{*}$, where the slope of $v=f\left(u\right)$ is larger than that of $v=g\left(u\right)$, i.e., $\frac{1+c{v}^{*}}{-c{u}^{*}}<\frac{r(1+a-2u^{*})}{1+2c{v}^{*}}$. Thus, $\det J_{E^{*}}>0$. Thus $E^{*}$ may be node or focus, and its stability depends on the trace

$$\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}).$$

(9)

Thus, we have the following conclusions on the stability of $E^{*}$.

Theorem 1.

If $c<\frac{1}{r(1-a)}$, then there exists a unique $p_H\in (1,\frac{2}{a+1})$ such that the unique interior equilibrium $E^{*}$ is locally asymptotically stable when $1<p<p_H$, and unstable when $p_H<p<\frac{1}{a}$. Moreover, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

Proof. 

Solving for $\mathrm{tr}{J}_{E^{*}}=0$, when $\frac{a+1}{2}<\frac{1}{p}<1$, we obtain a unique $u_H^{*}\in (\frac{a+1}{2},1)$, such that $\mathrm{tr}{J}_{E^{*}}>0$ on $(0,u_H^{*})$, and $\mathrm{tr}{J}_{E^{*}}<0$ on $(u_H^{*},1)$ (see Figure 2). Moreover, it is easy to verify that $\frac{\partial u^{*}}{\partial p}<0$; thus, corresponding to $u_H^{*}$, we obtain a unique $p_H$ ($1<p_H<\frac{2}{a+1}$), such that $\mathrm{tr}{J}_{E^{*}}>0$ when $p\in (p_H,\frac{1}{a})$, and $\mathrm{tr}{J}_{E^{*}}<0$ when $p\in (1,p_H)$.

Now, we verify the transversality condition. Let $\mu =\alpha \left(p\right)\pm i\omega \left(p\right)$ be the roots of ${\mu }^2-\mathrm{tr}{J}_{E^{*}}\mu +\det J_{E^{*}}=0$ when p is near $p_H$. We have ${\alpha }^{\prime }\left(p\right)=\frac{1}{2}\frac{\mathrm{d}\mathrm{tr}{J}_{E^{*}}}{\mathrm{d}p}=\frac{1}{2}[-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)-2r{u}^{*}{u}^{{*}^{\prime }}\left(p\right)$$-m{u}^{*}-mp{u}^{{*}^{\prime }}\left(p\right)]$, where $u^{*}\left(p\right)$ is a function of p determined by (6). Taking the derivative of both sides of (6) with respect to p, we have

$$-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)=\frac{(-u-p{u}^{{*}^{\prime }}\left(p\right))[c{p}^2{u}^{*2}+2cp{u}^{*}(1-p{u}^{*})]}{c^2{p}^4{u}^{*4}},$$

thus

$$-u-p{u}^{{*}^{\prime }}\left(p\right){\mid }_{p=p_H}=\frac{-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)c^2{p}^4{u}^{*4}}{c{p}^2{u}^{*2}+2cp{u}^{*}(1-p{u}^{*})}{\mid }_{p=p_H}>0,$$

as $\frac{a+1}{2}<u_H^{*}<\frac{1}{p}$ and $u^{{*}^{\prime }}\left(p\right)<0$. Thus, ${\alpha }^{\prime }\left(p\right){\mid }_{p=p_H}>0$. □

Remark 2.

When there is no cooperative hunting in predator, i.e., $c=0$, the v-nullcline is $u=\frac{1}{p}$, which is vertical. $\mathrm{tr}{J}_{E^{*}}{{|}_{c=0}=-\frac{r}{p}(\frac{2}{p}-a-1),\det J_{E^{*}}|}_{c=0}=m{v}^{*}.$ The system undergoes a Hopf bifurcation near $E^{*}$ when $\frac{1}{p}=\frac{a+1}{2}$, which is the top of the u-nullcline $v=r(1-u)(u-a)$. In fact, it has been discussed widely that the stability of positive equilibrium can be stated graphically by the u-nullcline when the v-nullcline is vertical: $(u^{*},v^{*})$ is unstable if the v-nullclines intersect to the left of a local maximum of the u-nullcline, and stable if they intersect to the right [29,58,59,60], and thus, Hopf bifurcation occurs at the “top of the hump” of u-nullcline. However, for system (7) in this paper, the v-nullcline is not vertical, and we can verify that the intersection of the u-nullcline and the v-nullcline corresponding to the Hopf bifurcation point $p=p_H$ is on the right of the “top of the hump" of u-nullcline. In fact, the “top of the hump” of u-nullcline $v=f\left(u\right)$ achieves at $u=\frac{a+1}{2}$, at which the corresponding value of p is denoted by $p_{\mathrm{top}}$. When $p=p_{\mathrm{top}}$, $E^{*}$ can be also proved to be unstable. Comparing the results in systems with and without cooperation, we can conclude that cooperative hunting is more likely to bring instability into $E^{*}$.

In order to determine the direction of Hopf bifurcation and the stability of the Hopf bifurcating periodic solution, we need to calculate the normal form near the Hopf bifurcation point. Through direct calculation following the steps in [61], we can get the following truncated normal form:

$$\begin{array}{c}\dot{r}={\alpha }^{\prime }\left(p_H\right)pr+a\left(p_H\right)r^3+O\left(r\right|p-p_H{|}^2,r^3|p-p_H|,r^5),\hfill \\ \dot{\theta }=\omega \left(p_H\right)+{\omega }^{\prime }\left(p_H\right)(p-p_H)+c\left(p_H\right)r^2+O\left(\right|p-p_H{|}^2,r^2|p-p_H|,r^4).\hfill \end{array}$$

Recalling that ${\alpha }^{\prime }\left(p_H\right)>0$, the direction of Hopf bifurcation and stability of Hopf bifurcating periodic solution are determined by the first Lyapunov coefficient $a\left(p_H\right)$, and we have the following conclusion.

Theorem 2.

If $c<\frac{1}{r(1-a)}$, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

 (i)

If $a\left(p_H\right)<0$, the bifurcating periodic solution is orbitally asymptotically stable, and it is bifurcating from $E^{*}$ as p increases and passes $p_H$.

 (ii)

If $a\left(p_H\right)>0$, the bifurcating periodic solution is unstable, and it is bifurcating from $E^{*}$ as p decreases and passes $p_H$.

3.1.2. The Global Dynamics of Kinetic System with Weak Cooperative Hunting

From the previous discussion, we have known the stability of all equilibria when $c<\frac{1}{r(1-a)}$. Motivated by the work of [29,30], we investigate the global dynamics of system (7). To this end, we should first study the properties of stable manifold of $E_a$ and unstable manifold of $E_1$, denoted by ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$, respectively. For the region bounded by ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$, we prove the existence or nonexistence of periodic orbits. We leave them in Appendix A, and only list main conclusions on the existence of a loop of heteroclinic orbit and global dynamics in the following.

From the monotonicity of S and U given in Proposition A1, we have the following conclusion.

Proposition 2.

If $c<\frac{1}{r(1-a)}$, then there exists a unique $p^{\#}\in (1,\frac{1}{a})$, such that ${\mathrm{\Gamma }}_{p^{\#}}^s={\mathrm{\Gamma }}_{p^{\#}}^u$, forming a heteroclinic orbit from $E_1$ to $E_a$.

In fact, there is another heteroclinic orbit from $E_a$ to $E_1$, which is formed by the unstable manifold of $E_a$ and the stable manifold of $E_1$ on the $u-$axis. Thus, there is a loop of heteroclinic orbits from $E_1$ to $E_a$, and then back to $E_1$.

Theorem 3.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

If $p\ge \frac{1}{a}$, $E_0(0,0)$ is globally asymptotically stable (see Figure 3a).

 (ii)

There is an ${\epsilon }_1>0$ such that if $\frac{1}{a}-{\epsilon }_1<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 3b).

 (iii)

There is an ${\epsilon }_2>0$ such that if $1<p<1+{\epsilon }_2$, then ${\mathrm{\Gamma }}_p^u$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E^{*}$ (see Figure 3c).

 (iv)

If $p<1$, the orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E_1$ (see Figure 3d).

We leave the proof in Appendix A.2.

Theorem 3 provides a global description of the dynamical behaviour of system (7) for $p\in {\mathbb{R}}^{+}\backslash (1+{\epsilon }_2,\frac{1}{a}-{\epsilon }_1)\stackrel{}{=}{\mathbb{R}}^{+}\backslash I$. When $p\in I$, if every periodic orbit of system (7) is orbitally stable, thus there can be at most one such orbit. From Proposition A5, we have the following conclusion.

Theorem 4.

Suppose that $c<\frac{1}{r(1-a)}$, the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable. Then, $p_H<p^{\#}$.

 (i)

If $1<p<p_H$, ${\mathrm{\Gamma }}_p^u$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E^{*}$.

 (ii)

If $p_H<p<p^{\#}$, $E^{*}$ is a repellor, and there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s$. The orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to the limit cycle. (see Figure 4a,b).

 (iii)

If $p=p^{\#}$, ${\mathrm{\Gamma }}_p^s={\mathrm{\Gamma }}_p^u$, there are two heteroclinic orbits forming a loop of heteroclinic orbits from $E_1$ to $E_a$ and back to $E_1$ (see Figure 4c). The orbits through any point exterior to the loop converge to $E_0$, and the orbits through any point interior to the cycle converge to the loop.

 (iv)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 4d).

Remark 3.

We leave the proof in Appendix A.3.

From the former conclusion on the global dynamics of model (7) with weak cooperation, we find that no matter how we choose the value of p, both of the two species are extinct if the ratio of predator to prey is high.

Now, we discuss the population behavior when the ratio of predator to prey is low. When $p<1$, because of the low conversion rate, the predator is extinct, while the prey reach the capacity of the environment. When $1<p<p_H$, the predator and prey coexist, and tend to a constant value. When $p_H<p<p^{\#}$, the predator and prey coexist, but oscillate sustainably. The amplitude of oscillation increases as p increases, and the minimum of the predator population is close to zero when p is close to $p^{\#}$, which will increase of risk on the extinction of predator. When the value of p is too high, high growth rate of prey leads to over hunting, and finally, both of the two species are extinct.

Remark 4.

Similar to the case in the model with Allee effect, there is a unique positive equilibrium in the model without Allee effect, and the varying p makes $E^{*}$ unstable and induces Hopf bifurcation. However, there is an additional saddle point $(a,0)$ in the model with Allee effect, and the stable manifold of the saddle point separates the positive quadrant into two positive invariant regions. If the initial populations are above the stable manifold of $(a,0)$, extinction happens for both species in the model with Allee effect, while coexistence happens for the model without Allee effect. This extinction is mediated by too high predator density that drives the prey density below the Allee threshold and both species become extinct.

3.2. Bifurcation and Global Dynamics of Kinetic System with Strong Cooperative Hunting

By strong cooperation we mean $c>\frac{1}{r(1-a)}$. In this section, we first analyze the stability of interior equilibrium, then we prove the existence of Hopf bifurcation and give the condition of the existence and nonexistence of loop of heteroclinic orbits. We also exhibit the complex dynamics of model (7), such as limit cycle, loop of heteroclinic orbits, and homoclinic cycle.

3.2.1. Stability and Hopf Bifurcation of Interior Equilibria

Now, we consider the stability of interior equilibria. For $E^{*}$ or $E_R^{*}$, the corresponding Jacobian matrix is

$$J=\left(\begin{array}{cc}ru(1+a-2u)& -2cuv-u\\ mpv(1+cv)& mpcuv\end{array}\right),$$

(10)

and thus

$$\begin{array}{c}\mathrm{tr}J=-ru(2u-a-1)+mpcuv=-ru(2u-a-1)+m(1-pu),\hfill \\ \det J=mpuv\left[rcu(1+a-2u)+(1+cv)(1+2cv)\right]\hfill \\ =\frac{mv}{u}\left[rcp{u}^3(1+a-2u)+\frac{2}{p}-u\right].\hfill \end{array}$$

If $\det J=0$, then $\frac{r(1+a-2u)}{1+2cv}=\frac{1+cv}{-cu}$, which means that $v=f\left(u\right)$ is tangent to $v=g\left(u\right)$. In fact, the point of tangency is $E_R^{**}$, thus $\det J_{E_R^{**}}=0$. It is clear that $\det J>0$ for $u<u_R^{**}$, and $\det J<0$ for $u>u_R^{**}$ (see Figure 5). The component $u^{*}$ of $E^{*}$ is always satisfying $u^{*}<u_R^{**}$, and the component $u_R^{*}$ of $E_R^{*}$ is satisfying $u_R^{*}>u_R^{**}$ (see Figure 1). It follows that for $E^{*}$, $\det J_{E^{*}}>0$, while for $E_R^{*}$, $\det J_{E_R^{*}}<0$. Thus, $E_R^{*}$ exists when $p_{SN}<p<1$, and it is a saddle. $E^{*}(u^{*},v^{*})$ exists when $p_{SN}<p<\frac{1}{a}$, and $\det J_{E^{*}}>0$. Thus, $E^{*}$ may be node or focus, and its stability depends on the trace

$$\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}).$$

(11)

Similar as the proof in Theorem 1, solving for $\mathrm{tr}{J}_{E^{*}}=0$, when $\frac{a+1}{2}<\frac{1}{p}<\frac{1}{p_{SN}}$, there is a unique $u_H^{*}\in (\frac{a+1}{2},\frac{1}{p_{SN}})$ such that $\mathrm{tr}{J}_{E^{*}}>0$ on $(0,u_H^{*})$, and $\mathrm{tr}{J}_{E^{*}}<0$ on $(u_H^{*},\frac{1}{p_{SN}})$ (see Figure 2). Recalling that $\frac{\partial u^{*}}{\partial p}<0$, corresponding to $u_H^{*}$, we obtain a unique $p_H$ ($p_{SN}<p_H<\frac{2}{a+1}$), such that $\mathrm{tr}{J}_{E^{*}}>0$ when $p\in (p_H,\frac{1}{a})$, and $\mathrm{tr}{J}_{E^{*}}<0$ when $p\in (p_{SN},p_H)$.

Let $\mu =\alpha \left(p\right)\pm i\omega \left(p\right)$ be the roots of ${\mu }^2-\mathrm{tr}{J}_{E^{*}}\mu +\det J_{E^{*}}=0$ when p is near $p_H$. We can prove ${\alpha }^{\prime }\left(p\right){\mid }_{p=p_H}>0$, in a similar way as in Theorem 1. Therefore, we can get the following theorem.

Theorem 5.

When $c>\frac{1}{r(1-a)}$, there exists a unique $p_H\in (p_{SN},\frac{2}{a+1})$ such that $E^{*}$ is locally asymptotically stable if $p_{SN}<p<p_H$, and unstable if $p_H<p<\frac{1}{a}$. Moreover, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

3.2.2. The Global Dynamics of Kinetic System with Strong Cooperative Hunting

When $c>\frac{1}{r(1-a)}$, system (7) may have one or two interior equilibria, which brings a few complications to the dynamics of model (7). There are three different cases of global dynamics when c takes different sizes. In this section, we exhibit the dynamics combining theoretical analysis and numerical simulations.

First, we consider the properties of the stable manifold of $E_a$, unstable manifold of $E_1$, as well as stable and unstable manifold of $E_R^{*}$, which are in Appendix B. Then, we can determine the global dynamics of model (7) when p is chosen in the following ranges. As in Theorem 3, we can prove the following conclusion.

Theorem 6.

Suppose that $c>\frac{1}{r(1-a)}$.

 (i)

If $p\ge \frac{1}{a}$, $E_0(0,0)$ is globally asymptotically stable (see Figure 6a).

 (ii)

If $p<\frac{1}{a}$ and near $\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 6b).

 (iii)

If $p<p_{SN}$, the orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_1$ (see Figure 6c).

The remaining question is how are the dynamics of system (7) for p chosen in the rest ranges, i.e., $p_{SN}<p<\frac{1}{a}$. As larger c means less steep of the v-nullcline, the critical points (for example, $p_H$, $p^{\#}$) may appear in different ranges, and there are the following three different cases.

Case 1 Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H>1$. From Theorem A1, there is a loop of heteroclinic orbits when $p=p^{\#}$. We have the following conclusion.

Theorem 7.

Suppose that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H>1$. Assume that the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable, then $p_H<p^{\#}$.

 (i)

If $p_{SN}<p<1$, the upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ and exterior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E_1$ (see Figure 7a).

 (ii)

If $1<p<p_H$, ${\mathrm{\Gamma }}_p^u\left(E_1\right)$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E^{*}$.

 (iii)

When $p_H<p<p^{\#}$, $E^{*}$ is a repellor, and there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s\left(E_a\right)$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to the limit cycle (see Figure 7b,c).

 (iv)

When $p=p^{\#}$, ${\mathrm{\Gamma }}_{p^{\#}}^s\left(E_a\right)={\mathrm{\Gamma }}_{p^{\#}}^u\left(E_1\right)$, and there are two heteroclinic orbits forming a loop of heteroclinic orbits between $E_1$ and $E_a$ (see Figure 7d). The orbits through any point exterior to the cycle converge to $E_0$, and the orbits through any point interior to the cycle converge to the cycle.

 (v)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (phase portrait is similar as Figure 6b).

Proof. 

From Proposition A6, $S_a\left(p\right)$ is decreasing with p, combining with $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, then ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ enters $E_a$ from the region $\left\{\right(u,v):v>f(u\left)\right\}$ for all $p<1$. As $E^{*}$ is stable when $p<p_H$, then $E^{*}$ is stable for all $p_{SN}<p<1$. If there is a periodic orbit below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, it must encircle $E^{*}$, and it is unstable from inside, which contradicts with our assumption of the orbital stability of periodic orbit. It means that there is no periodic orbit under ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ for all $p<1$. From Proposition A9, $E^{*}$ is the $\omega -$limit set of ${\mathrm{\Omega }}_{22}$, and ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ must be above ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$. The upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$ (see Figure 7a). Thus, the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The proof of (ii)–(v) is similar as that of Theorem 4. □

Case 2 Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H<1$. From Theorem A1, there is a loop of heteroclinic orbits when $p=p^{\#}$.

Theorem 8.

Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H<1$. Assume that the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable, then $p_H<p^{\#}$.

 (i)

If $p_{SN}<p<p_H$, the upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$, and the downward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E_1$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ and exterior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E_1$ (phase portrait is similar as Figure 7a).

 (ii)

When $p_H<p<1$, there is a unique limit cycle inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ of $E_R^{*}$ (see Figure 8a). The orbits through any point interior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to the limit cycle.

 (iii)

When $1<p<p^{\#}$, there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s$. The orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to the limit cycle (see Figure 8b).

 (iv)

When $p=p^{\#}$, ${\mathrm{\Gamma }}_{p^{\#}}^s\left(E_a\right)={\mathrm{\Gamma }}_{p^{\#}}^u\left(E_1\right)$, there are two heteroclinic orbits forming a loop of heteroclinic orbits from $E_1$ to $E_a$ and back to $E_1$ (see Figure 8c). The orbits through any point exterior to the cycle converge to $E_0$, and the orbits through any point interior to the cycle converge to the cycle.

 (v)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (phase portrait is similar as Figure 6b).

Proof. 

We can prove (i) similar as in Theorem 7. (ii) As $a\left(p_H\right)<0$ and ${\alpha }^{\prime }\left(p_H\right)>0$, there is a stable periodic orbit bifurcating from Hopf bifurcation when $p>p_H$. As in the proof of Theorem 7, we can prove that ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ must be above ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$. From Proposition A9, the unique limit cycle is the $\omega -$limit set of ${\mathrm{\Omega }}_{22}$. The proof of (iii)–(v) is similar as that of Theorem 4. □

Case 3 Choose c such that $c>\frac{1}{r(1-a)}$ and $U_1\left(1^{+}\right)>S_a\left(1^{+}\right)$. From Theorem A1, there is no heteroclinic orbits from $E_1$ to $E_a$. It is easy to prove that when $1<p<\frac{1}{a}$, $E_0$ is globally asymptotically stable (see Figure 9e).

Remark 5.

In this case, the location of ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ and ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ are very complicated. We can observe the following dynamics by numerical simulations.

 (i)

Loop of heteroclinic orbits. The downward branch of the unstable manifold of $E_R^{*}$ connects $E_R^{*}$ to $E_1$, and the upward connects $E_R^{*}$ to $E_a$, which collides with the the stable manifold of $E_a$. The upward and downward unstable manifold of $E_R^{*}$, together with the unstable manifold of $E_a$ on the $u-$ axis, forms a loop of heteroclinic orbits among $E_R^{*}$, $E_a$ and $E_1$ when $p={\overline{p}}^{\#}$ (see Figure 9d).

 (ii)

Homoclinic cycle. The upward unstable manifold and the left stable manifold of $E_R^{*}$ collide, which forms a homoclinic cycle. Denote the parameter p as $p=p_{hom}$ (see Figure 9c).

 (iii)

Limit cycle induced by Hopf bifurcation (see Figure 9a,b).

Remark 6.

There are two positive equilibria $E^{*}$ and $E_R^{*}$ in both of the model with or without Allee effect, where $E_R^{*}$ is always a saddle. In the model without Allee effect, the stable manifold of $E_R^{*}$ separates the positive quadrant into two positive invariant regions. Two species coexist if the initial populations are above the stable manifold. However, there is an additional stable manifold of saddle point $(a,0)$ in the model with Allee effect, and two stable manifolds of the two saddle points separate the positive quadrant into three positive invariant regions. Two species only coexist between the two stable manifolds, and both species go extinct if the initial populations is above the stable manifold of $(a,0)$. This means that the region of coexistence of two species is reduced due to the Allee effect.

4. Diffusion-Driven Turing Instability and Turing–Hopf Bifurcation

In the previous section, we have discussed both local bifurcation and global dynamics of kinetic model (7). In this section, we consider the effect of the diffusion on the stability of the interior constant steady states; thus, we set ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$ in system (4). If an equilibrium is linearly stable in the absence of diffusion, and it becomes unstable in the presence of diffusion, we call such an instability Turing instability. As $E_R^{*}$ is always a saddle if it exists, thus Turing instability can only happen near $E^{*}$ in both cases: weak cooperation and strong cooperation. We first investigate the existence of Turing instability, then we consider Turing–Hopf bifurcation near $E^{*}$.

For Neumann boundary condition, we define the real-valued Sobolev space

$$X=\{{(u,v)}^T\in H^2(0,l\pi )\times H^2(0,l\pi ), \frac{\partial u}{\partial x}=\frac{\partial v}{\partial x}=0,\mathrm{at} x=0,l\pi \}.$$

(12)

The linearization of system (4) with ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$ at the constant steady state $E^{*}(u^{*},v^{*})$ is given by

$$\left(\begin{array}{c}\frac{\partial u}{\partial t}\\ \frac{\partial v}{\partial t}\end{array}\right)=D\left(\begin{array}{c}\Delta u\\ \Delta v\end{array}\right)+A\left(\begin{array}{c}u\\ v\end{array}\right)\stackrel{∆}{=}L\left(\begin{array}{c}u\\ v\end{array}\right),$$

(13)

where

$$D=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right), A=\left(\begin{array}{cc}r{u}^{*}(1+a-2u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ mp(1+c{v}^{*})v^{*}& mpc{u}^{*}{v}^{*}\end{array}\right).$$

It is well known that the eigenvalues of $D\Delta$ on X are $-d_1\frac{n^2}{l^2}$ and $-d_2\frac{n^2}{l^2}$, $k\in {\mathbb{N}}_0=\{0,1,2,...\}$. From straightforward calculation, we obtain the characteristic equations

$${\Delta }_n={\lambda }^2-T_n\lambda +J_n=0, n=0,1,2,...,$$

(14)

where

$$\begin{array}{c}{T}_n=-\left(d_1\frac{n^2}{l^2}+d_2\frac{n^2}{l^2}+r{u}^{*}(2u^{*}-a-1)-mpc{u}^{*}{v}^{*}\right)\hfill \\ =-d_1\frac{n^2}{l^2}-d_2\frac{n^2}{l^2}+\mathrm{tr}{J}_{E^{*}},\hfill \\ J_n=d_1{d}_2\frac{n^4}{l^4}-r{u}^{*}(1+a-2u^{*})d_2\frac{n^2}{l^2}-mpc{u}^{*}{v}^{*}{d}_1\frac{n^2}{l^2}+\det J_{E^{*}}.\hfill \end{array}$$

(15)

From the previous section, in the absence of diffusion, i.e., $d_1=d_2=0$, $E^{*}$ is asymptotically stable if $1<p<p_H$ in the case of $c<\frac{1}{r(1-a)}$ (if $p_{SN}<p<p_H$ in the case of $c>\frac{1}{r(1-a)}$). If there exists an $n\in \mathbb{N}$, such that ${\Delta }_n=0$ has roots with positive real part when $p<p_H$, Turing instability occurs. As $\mathrm{tr}{J}_{E^{*}}<0$ when $p<p_H$, and $\det J_{E^{*}}>0$ always satisfies, from (15), we have $T_n<0$ for any n when $p<p_H$. Therefore, when $p<p_H$, the signs of the real parts of roots of (14) are determined by the signs of $J_n$.

In fact, the curve of $J_n=0$ is a hyperbola on $d_1-d_2$ plane, whose horizontal and vertical asymptotes are $d_2=\frac{mpc{u}^{*}{v}^{*}{l}^2}{n^2}$ and $d_1=-\frac{r{u}^{*}(2u^{*}-1-a)l^2}{n^2}$. We only need to consider the right branch of the hyperbola, which intersects with the $d_1-$axis at $d_1=\frac{\det J_{E^{*}}{l}^2}{mpc{u}^{*}{v}^{*}{n}^2}$. Thus, with respect to n, both the horizontal asymptote of the right branch of the hyperbola and the intersection with the $d_1-$axis are decreasing.

Denote

$$d_2^T(n,d_1)\stackrel{∆}{=}\frac{mpc{u}^{*}{v}^{*}{d}_1\frac{n^2}{l^2}-\det J_{E^{*}}}{d_1\frac{n^4}{l^4}+r{u}^{*}(2u^{*}-1-a)\frac{n^2}{l^2}}.$$

(16)

Noting that $u^{*}>\frac{a+1}{2}$ when $p\le p_H$, we have $J_n<0$ if $d_2<d_2^T(n,d_1)$, and $J_n>0$ if $d_2>d_2^T(n,d_1)$.

Theorem 9.

If $c<\frac{1}{r(1-a)}$, suppose $1<p<p_H$ (If $c>\frac{1}{r(1-a)}$, suppose $p_{SN}<p<p_H$). The constant steady state $E^{*}$ is locally asymptotically stable if $d_2>d_2^T(n,d_1)$ for all $n\in \mathbb{N}$, while it is Turing unstable if $d_2<d_2^T(n,d_1)$ for some $n\in \mathbb{N}$ satisfying $n>l\sqrt{\frac{\det J_{E^{*}}}{mpc{u}^{*}{v}^{*}{d}_1}}$.

On the $d_1-d_2$ plane, we call the boundary curve of stable region of steady state $E^{*}$ the Turing bifurcation curve $l^T$, which is formed by a sequence of curve segments $l_n^T$$(n=1,2,...)$

$$d_2=d_2^T(n,d_1), \mathrm{for} d_{1,n}<d_1\le d_{1,n-1},$$

(17)

where $d_{1,n}$ is the intersection of $d_2=d_2^T(n+1,d_1)$ and $d_2=d_2^T(n,d_1)$, and $d_{1,0}$ can be infinite.

When $p=p_H$, $T_n<0$ for all $n\in \mathbb{N}$, similar as the analysis in the case of $p<p_H$, we can also get a Turing bifurcation curve, and we have the following conclusion.

Remark 7.

Assume, in the $p-d_2$ plane, $d_2=d_2^T(n,d_1)$ and $p=p_H$ intersect at a point $TH$, then the characteristic equation has a pair of purely imaginary roots and a zero root at $TH$. According to the general theory of [62], a Turing–Hopf bifurcation may appear. However, rigorous derivation for the normal form is impossible due to the implicit form of $E^{*}$ in this paper. We shall give some numerical illustrations to show the dynamics of the system near the Turing–Hopf bifurcation point.

5. The Dynamics of Diffusive System with Two Delays

In the previous section, we have found the conditions of stability and Turing instability of constant steady state $E^{*}$ of system (4) when ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$. We know that if $p<p_H$ and $d_2>d_2^T(n,d_1)$ for all $n\in \mathbb{N}$, the steady state $E^{*}$ is locally asymptotically stable. In this section, we investigate the diffusive system (4) under the assumption $p<p_H$ and $d_2>d_2^T(n,d_1)$, and focus on the effect of two delays on the dynamics of the diffusive system near $E^{*}$.

5.1. Hopf and Double Hopf Bifurcation Induced by Two Delays

In this subsection, we investigate the existence of Hopf bifurcation induced by two delays by the method of stability switching curves given in [63], and give the condition of the existence of double Hopf bifurcation.

The linearization of system (4) at the steady state $E^{*}(u^{*},v^{*})$ is given by

$$\frac{\partial }{\partial t}\left(\begin{array}{c}u(x,t)\hfill \\ v(x,t)\hfill \end{array}\right)=(D\Delta +A)\left(\begin{array}{c}u(x,t)\hfill \\ v(x,t)\hfill \end{array}\right)+B\left(\begin{array}{c}u(x,t-{\mathrm{\tau }}_1)\hfill \\ v(x,t-{\mathrm{\tau }}_1)\hfill \end{array}\right)+C\left(\begin{array}{c}u(x,t-{\mathrm{\tau }}_2)\hfill \\ v(x,t-{\mathrm{\tau }}_2)\hfill \end{array}\right),$$

(18)

where

$$\begin{array}{c}A=\left(\begin{array}{cc}r{u}^{*}(1-u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ 0& 0\end{array}\right), B=\left(\begin{array}{cc}-r{u}^{*}(u^{*}-a)& 0\\ 0& 0\end{array}\right),\hfill \\ C=\left(\begin{array}{cc}0& 0\\ mp(1+c{v}^{*})v^{*}& mpc{u}^{*}{v}^{*}\end{array}\right),D=\mathrm{diag}(d_1,d_2).\hfill \end{array}$$

(19)

$u(x,t)$ and $v(x,t)$ satisfy the homogeneous Neumann boundary condition.

From Wu [64], the corresponding characteristic equation of (18) is

$$\det \left(\lambda I-M_n-A-B{e}^{-\lambda {\mathrm{\tau }}_1}-C{e}^{-\lambda {\mathrm{\tau }}_2}\right)=0,$$

(20)

where I is a $2\times 2$ identity matrix, $M_n=-n^2/l^2\mathrm{diag}(d_1,d_2)$, $n\in {\mathbb{N}}_0=\{0,1,2,...\}$. Equation (20) can be written in the following form:

$$D_n(\lambda ;{\mathrm{\tau }}_1,{\mathrm{\tau }}_2)=P_{0,n}\left(\lambda \right)+P_{1,n}\left(\lambda \right)e^{-\lambda {\mathrm{\tau }}_1}+P_{2,n}\left(\lambda \right)e^{-\lambda {\mathrm{\tau }}_2}+P_{3,n}\left(\lambda \right)e^{-\lambda ({\mathrm{\tau }}_1+{\mathrm{\tau }}_2)}=0,$$

(21)

where

$$\begin{array}{c}{P}_{0,n}\left(\lambda \right)=(\lambda +d_1\frac{n^2}{l^2}-a_{11})(\lambda +d_2\frac{n^2}{l^2}),\hfill \\ P_{1,n}\left(\lambda \right)=-b_{11}(\lambda +d_2\frac{n^2}{l^2}),\hfill \\ P_{2,n}\left(\lambda \right)=-c_{22}(\lambda +d_1\frac{n^2}{l^2}-a_{11})-a_{12}{c}_{21},\hfill \\ P_{3,n}\left(\lambda \right)=b_{11}{c}_{22}.\hfill \end{array}$$

(22)

The characteristic equation with the form of (21) has been investigated by Lin and Wang [63]. They derived an explicit expression for the stability switching curves, on which there is a pair of purely imaginary roots for (21). Moreover, they gave a criterion to determine the crossing directions, i.e., on which side of the stability switching curve there are two more characteristic roots with positive real parts. Using this method, we can find all the stability switching curves in the $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ plane and determine the crossing directions. We leave the details in Appendix C and Appendix D.

Moreover, we have the following Hopf bifurcation theorem with two parameters.

Theorem 10.

For each $j\in \{1,2,\cdots ,N\}$, ${\mathcal{T}}_n^j$, defined by (A9), is a Hopf bifurcation curve in the following sense: for any $p\in {\mathcal{T}}_n^j$ and for any smooth curve Γ intersecting with ${\mathcal{T}}_n^j$ transversely at p, we define the tangent of Γ at p by $\overrightarrow{l}$. If $\frac{\partial \mathrm{Re}\lambda }{\partial \overrightarrow{l}}{\mid }_p\ne 0$, and the other eigenvalues of (21) at p have non-zero real parts, then system (4) undergoes a Hopf bifurcation at p when parameters $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross ${\mathcal{T}}_n^j$ at p along Γ.

The theorem can be proved by a similar method in [49].

Remark 8.

Suppose that ${\mathcal{T}}_{k_1}^{j_1}$ and ${\mathcal{T}}_{k_2}^{j_2}$ intersect at a point $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$, with the corresponding values of ω being ${\omega }_{j_1,k_1}\in {\mathrm{\Omega }}_{j_1,k_1}$ and ${\omega }_{j_2,k_2}\in {\mathrm{\Omega }}_{j_2,k_2}$. Then, there are two pairs of purely imaginary roots of (21) at the intersection, which are $\pm i{\omega }_{j_1,k_1}$ and $\pm i{\omega }_{j_2,k_2}$, denoted by $\pm i{\omega }_1$ and $\pm i{\omega }_2$ for convenience. Thus, system (4) may undergo double Hopf bifurcations at the intersection of two stability switching curves near the constant steady state $E^{*}$.

5.2. Normal Form on the Center Manifold for Double Hopf Bifurcation

In this subsection, applying the normal form method of partial functional differential equations [65], we derive normal form of double Hopf bifurcation taking two delays as bifurcation parameters. Then, we can classify the topological structures of various bifurcating solutions, and get the dynamics in the neighborhood of the double Hopf bifurcation point in system (4).

For the real-valued Sobolev space X defined in (12), define the complexification space of the real-valued Hilbert space X by

$$X_{\mathbb{C}}:=X\oplus iX=\{U_1+i{U}_2:U_1,U_2\in X\},$$

with the general complex-value $L^2$ inner product

$$\langle U,V\rangle ={\int }_0^{l\pi }({\overline{u}}_1{v}_1+{\overline{u}}_2{v}_2)dx, \mathrm{f}\mathrm{o}\mathrm{r} U={(u_1,u_2)}^T,V={(v_1,v_2)}^T\in X_{\mathbb{C}}.$$

Let $\mathcal{C}:=C([-1,0],X_{\mathbb{C}})$ denote the phase space with the sup norm, and write $u^t\in \mathcal{C}$ for $u^t\left(\theta \right)=u(t+\theta )$, $-1\le \theta \le 0$.

Without loss of generality, we assume ${\mathrm{\tau }}_1<{\mathrm{\tau }}_2$ in this section. Denote the double Hopf bifurcation point by $({\mathrm{\tau }}_1^{*},{\mathrm{\tau }}_2^{*})$. Let $\overline{u}(x,t)=u(x,{\mathrm{\tau }}_{2}t)-u^{*},\overline{v}(x,t)=v(x,{\mathrm{\tau }}_{2}t)-v^{*}$, and set ${\sigma }_1={\mathrm{\tau }}_1-{\mathrm{\tau }}_1^{*}$ and ${\sigma }_2={\mathrm{\tau }}_2-{\mathrm{\tau }}_2^{*}$ as two bifurcation parameters. Drop the bars, and denote $U\left(t\right)={(u\left(t\right),v\left(t\right))}^T$, then system (4) can be written as

$$\frac{dU\left(t\right)}{dt}=D({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\Delta U\left(t\right)+L({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\left(U^t\right)+F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,U^t),$$

(23)

where

$$\begin{array}{c}D({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)=({\mathrm{\tau }}_2^{*}+{\sigma }_2)D={\mathrm{\tau }}_2^{*}D+{\sigma }_{2}D,\hfill \\ L({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\left(\varphi \right)=({\mathrm{\tau }}_2^{*}+{\sigma }_2)\left[A\varphi \left(0\right)+B\varphi (-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})+C\varphi (-1))\right],\hfill \\ F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,\varphi )=({\mathrm{\tau }}_2^{*}+{\sigma }_2){(F_1,F_2)}^T,\hfill \end{array}$$

with $A, B,C$ and $D$ being defined in (19), and

$$\begin{array}{c}{F}_1=r(1-u^{*}){\varphi }_1^2\left(0\right)-(1+2c{v}^{*}){\varphi }_1\left(0\right){\varphi }_2\left(0\right)+(ra-2r{u}^{*}){\varphi }_1\left(0\right){\varphi }_1(-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})\hfill \\ -c{u}^{*}{\varphi }_2^2\left(0\right)-r{\varphi }_1^2\left(0\right){\varphi }_1(-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})-c{\varphi }_1\left(0\right){\varphi }_2^2\left(0\right),\hfill \end{array}$$

$$\begin{array}{c}{F}_2=mp(1+c{v}^{*}){\varphi }_2\left(0\right){\varphi }_1(-1)+mpc[u^{*}{\varphi }_2\left(0\right)+v^{*}{\varphi }_1(-1)+{\varphi }_2\left(0\right){\varphi }_1(-1)]{\varphi }_2(-1).\hfill \end{array}$$

Consider the linearized system of (23)

$$\frac{dU\left(t\right)}{dt}={\mathrm{\tau }}_2^{*}D\Delta U\left(t\right)+{\mathrm{\tau }}_2^{*}(A{U}^t\left(0\right)+B{U}^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*})+C{U}^t(-1))\stackrel{}{=}{D}_0\Delta U\left(t\right)+L_0\left(U^t\right).$$

(24)

We know that the normalized eigenfunctions of $D\Delta$ on X corresponding to the eigenvalues $-d_1\frac{n^2}{l^2}$ and $-d_2\frac{n^2}{l^2}$( $n\in {\mathbb{N}}_0=\{0,1,2,\cdots \}$) are

$${\beta }_n^1\left(x\right)={\gamma }_n\left(x\right){(1,0)}^T \mathrm{and} {\beta }_n^2\left(x\right)={\gamma }_n\left(x\right){(0,1)}^T,$$

respectively, where ${\gamma }_n\left(x\right)=\frac{cos\frac{n}{l}x}{\Vert cos\frac{n}{l}{x\Vert }_{L^2}}$. Define ${\mathcal{B}}_n:=\mathrm{span}\left\{\langle v(·),{\beta }_n^j\rangle {\beta }_n^j | v\in \mathcal{C},j=1,2\right\}$, satisfying $L\left({\mathcal{B}}_n\right)({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)\subset \mathrm{span}\{{\beta }_n^1,{\beta }_n^2\}$. Denote $〈v(·),{\beta }_n〉={\left(\langle v(·),{\beta }_n^1\rangle ,\langle v(·),{\beta }_n^2\rangle \right)}^T.$

Write system (23) as

$$\frac{dU\left(t\right)}{dt}=D_0\Delta U\left(t\right)+L_0\left(U^t\right)+\tilde{F}(\sigma ,U^t),$$

(25)

where

$$\begin{array}{c}\hfill \tilde{F}(\sigma ,U^t)={\sigma }_2(D\Delta U^t\left(0\right)+A{U}^t\left(0\right)+B{U}^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*})+C{U}^t(-1))\\ \hfill +({\mathrm{\tau }}_2^{*}+{\sigma }_2)B(U^t(-({\mathrm{\tau }}_1^{*}+{\sigma }_1)/({\mathrm{\tau }}_2^{*}+{\sigma }_2))-U^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*}))+F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,U^t).\end{array}$$

System (25) can be rewritten as an abstract ordinary differential equation on $\mathcal{C}$ [65]

$$\frac{d}{dt}{U}^t=\mathcal{A}{U}^t+X_0\tilde{F}(\sigma ,U^t),$$

(26)

where $\mathcal{A}$ is the infinitesimal generator of the $C_0$-semigroup of solution maps of the linear equation (23), defined by

$$\mathcal{A}\phi =\dot{\phi }+X_0[D_0\Delta \phi \left(0\right)+L_0\left(\phi \right)-\dot{\phi }\left(0\right)],$$

(27)

with $\mathrm{dom}\left(\mathcal{A}\right)=\{\phi \in \mathcal{C}:\dot{\phi }\in \mathcal{C},\phi \left(0\right)\in \mathrm{dom}(\Delta )\}$, and $X_0$ is given by $X_0\left(\theta \right)=0$ for $\theta \in [-1,0)$ and $X_0\left(0\right)=I$. Clearly, $\mathcal{A}:{\mathcal{C}}_0^1\cap \mathcal{C}\to \mathcal{C}.$

Then, on ${\mathcal{B}}_n$, the linear equation

$$\frac{d}{dt}U\left(t\right)=D_0\Delta U\left(t\right)+L_0\left(U^t\right)$$

is equivalent to the retarded functional differential equation on ${\mathbb{C}}^2$:

$$\dot{z}\left(t\right)=-\frac{n^2}{l^2}{D}_{0}z\left(t\right)+L_0{z}^t.$$

(28)

By the Riesz representation theorem, there exists a matrix whose components are bounded variation functions ${\eta }_k\in BV([-1,0],{\mathbb{R}}^{2\times 2})$ such that

$$-\frac{n_k^2}{l^2}{D}_0\phi \left(0\right)+L_0\left(\phi \right)={\int }_{-1}^{0}d{\eta }_k\left(\theta \right)\phi \left(\theta \right),\phi \in \mathcal{C}.$$

Let $A_k$ ($k=1,2$) denote the infinitesimal generator of the semigroup generated by (28), and $A_k^{*}$ denotes the formal adjoint of $A_k$ under the bilinear form

$${(\alpha ,\beta )}_k=\alpha \left(0\right)\beta \left(0\right)-{\int }_{-1}^0{\int }_0^{\theta }\alpha (\xi -\theta )d{\eta }_k\left(\theta \right)\beta \left(\xi \right)d\xi .$$

The calculations of normal form are very long, so we leave them in supplementary materials. Based on the derivation in the Supplementary Materials, the normal form truncated to the third order on the center manifold for double Hopf bifurcation is obtained. Making the polar coordinate transformation, then we obtain the following system corresponding to the truncated normal form

$$\begin{array}{c}{\dot{\rho }}_1=r_1({\nu }_1+r_1^2+b{r}_2^2),\hfill \\ {\dot{\rho }}_2=r_2({\nu }_2+c{r}_1^2+d{r}_2^2),\hfill \end{array}$$

(29)

where

$${\nu }_1={ϵ}_1(\mathrm{Re}{B}_{11}{\sigma }_1+\mathrm{Re}{B}_{21}{\sigma }_2),{\nu }_2={ϵ}_1(\mathrm{Re}{B}_{13}{\sigma }_1+\mathrm{Re}{B}_{23}{\sigma }_2),$$

$$b=\frac{{ϵ}_1{ϵ}_2\mathrm{Re}{B}_{1011}}{\mathrm{Re}{B}_{0021}},c=\frac{\mathrm{Re}{B}_{1110}}{\mathrm{Re}{B}_{2100}},d={ϵ}_1{ϵ}_2,$$

and $B_{11},B_{21},B_{13},B_{23},B_{1011},B_{0021},B_{1110}$ and $B_{2100}$ are the coefficients of the normal form obtained in Supplementary Materials. From Chapter 7.5 in [62], there are twelve distinct kinds of unfoldings for (29).

6. Numerical Simulations

In this section, we carry out some numerical simulations for the kinetic system (7), diffusive system without delays, and the diffusive system (4) with two delays, respectively. These numerical simulations confirm and illustrate our main theoretical results.

6.1. Numerical Simulations for Kinetic System

In this subsection, we carry out some numerical simulations for system (7). We fix the parameters as

$$a=0.23,r=1.1,m=0.31.$$

(30)

6.1.1. Numerical Simulations for Kinetic System with Weak Cooperative Hunting

For system (7), choose $c=0.25$ such that $c<\frac{1}{r(1-a)}$. To show the complex dynamics of system (7), different values of p are chosen, listed in

Table 1. Values of parameter p chosen in Figure 3 and Figure 4.

Figure	Figure 3a	Figure 3b	Figure 3c	Figure 3d
p	5.1	3.2	1.2	0.9
Figure	Figure 4a	Figure 4b	Figure 4c	Figure 4d
p	1.55	1.645	1.6491	1.95

and

Table 2. Values of parameter p chosen for showing case 1 in Figure 7 and Figure 8.

Figure	Figure 7a	Figure 7b	Figure 7c	Figure 7d
p	0.96	1.105	1.2	1.2068
Figure	Figure 8a	Figure 8b	Figure 8c
p	0.965	1.052	1.05665

. The corresponding phase portraits for different values of p have been illustrated in Figure 3 and Figure 4, which is drawn by pplane8 [66].

Using the method in [67], we can get Hopf bifurcation point $p_H=1.5432$, and the first Lyapunov coefficient is $-1.1211$, which means that the bifurcating periodic solution is asymptotically stable, and it is bifurcating from $E^{*}$ as p increases past $p_H$ from Theorem 2. We can draw the bifurcation diagram in Figure 10. If $p<p_H=1.5432$, $E^{*}$ is asymptotically stable (the black curve in Figure 10a). The increase of p induces instability to $E^{*}$. When $p>1.5432$, $E^{*}$ becomes unstable (the red curve in Figure 10a), and a unique limit cycle appears (marked by LPC). The period of limit cycle is increasing as p increases from $p_H$, which tends to infinite as $p\to 1.6491$ (see Figure 10b). When $p=p^{\#}=1.6491$, the limit cycle coincides with a loop of heteroclinic orbits. When $p>p^{\#}=1.6491$, the loop of heteroclinic orbits breaks and the limit cycle disappears. The corresponding phase portrait has been drawn in Figure 4, and the results are consistent with Theorem 4. Moreover, the amplitude of oscillation increases as p increases, and the minimum of the predator population is close to zero when p is close to $p^{\#}=1.6491$, which will increase the risk on the extinction of predator.

6.1.2. Numerical Simulations for Kinetic System with Strong Cooperative Hunting

In Section 3.2, we discuss three different cases of dynamics when c is chosen as different values. To show case 1, we fix $c=3$ and vary p as a bifurcation parameter. We get $p_H=1.1024$, and the first Lyapunov coefficient is $-2.3280$. Choosing different values of p listed in

Table 3. Values of parameter p chosen for showing case 3 in Figure 9.

Figure	Figure 9a	Figure 9b	Figure 9c	Figure 9d	Figure 9e
p	0.834	0.88	0.8919	0.91564	1.1

, we have drawn the corresponding phase portraits for different values of p in Figure 7.

To show case 2, choosing $c=5$ and varying p, we can get $p_H=0.9608$, and the first Lyapunov coefficient is $-2.9553$. Choosing p as listed in Table 2, the corresponding phase portraits have been illustrated in Figure 8.

To show case 3, choosing $c=8$ and varying p, we can get $p_{SN}=0.7665$, $p_H=0.8306$, and the first Lyapunov coefficient is $-3.732$. We can draw the bifurcation diagram in Figure 11. When $p<0.7665$ (i.e., on the left of LP), there is no interior equilibria. When $p>0.7665$ (i.e., on the right of LP), there are two equilibria $E_{*}$ (the black curve) and $E_R^{*}$ (the blue dotted curve). If $0.7665<p<0.8306$, $E^{*}$ is asymptotically stable (the blue curve). The increase of p induces instability to $E^{*}$. When $p>0.8306$, $E^{*}$ becomes unstable (the red curve), and a unique limit cycle appears (marked by LPC in Figure 11a). When p increasing from $p_H$, the period of limit cycle is increasing, and it tends to infinite as $p\to 0.8919$ (see Figure 11b). Moreover, the amplitude of oscillation increases as p increases, and the amplitude line touches the curve of $v-$value of $E_R^{*}$. It means that there is a cycle goes through $E_R^{*}$, which forms a homoclinic cycle. The limit cycle coincides with a homoclinic cycle when $p=p_{hom}=0.8919$. When $p>0.8919$, the homoclinic cycle breaks, and the limit cycle disappears. Choosing p as listed in Table 3, the corresponding phase portraits have been illustrated in Figure 9.

6.2. Numerical Simulations for Diffusive System without Delays

In this section, we carry out some numerical simulations for diffusive system (4) with ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$. We fix the parameters as

$$a=0.23,r=1.1,m=0.31,l=2.$$

(31)

To illustrate the dynamics of (4) with no delays in the case of weak cooperation, we choose $c=0.25$ such that $c<\frac{1}{r(1-a)}$. Let $p=1.4<p_H=1.5432$, from the previous section, $E^{*}(0.6882,0.1516)$ is stable when $d_1=d_2=0$. On $d_1-d_2$ plane, we can draw curves determined by (16) for $n\in \mathbb{N}$, and we only illustrate four curves for $n=2,3,4,5$. Correspondingly, from (17), we can get four curve segments, forming partial Turing bifurcation curve $l^T$ (see Figure 12a). From Theorem 9, if we choose $d_1$ and $d_2$ in the region above the curve, the constant steady state $E^{*}$ is stable. If we choose $d_1=1.496$ and $d_2=0.000688$ in the region under $l^T$, the steady state $E^{*}$ of the diffusive system is Turing unstable, and there is a spatially inhomogeneous steady state (see Figure 13).

If we fix $d_1=1.496$, from (16), we can draw Turing bifurcation curve on $p-d_2$ plane (see Figure 12b), and $d_2=d_2^T(5,d_1)$ intersects line $p=p_H$ at $TH (1.5432,0.0009)$. Choosing $p=1.5432,d_2=0.00081$ near $TH$, two stable spatially inhomogeneous periodic solutions coexist when we choose two different initial values (see Figure 14).

To illustrate the dynamics of (4) without delays in the case of strong cooperation, we choose $c=8$ such that $c>\frac{1}{r(1-a)}$. Let $p=0.8<p_H=0.8306$. From (16) and (17), we can draw partial Turing bifurcation curve $l^T$ on $d_1-d_2$ plane (see Figure 15a). The constant steady state $E^{*}(0.7392,0.0864)$ is stable when we choose $d_1$ and $d_2$ in the region above $l^T$.

If we choose $d_1=0.13$ and $d_2=0.006$ in the region below $l^T$, from Theorem 9, the steady state $E^{*}$ of the diffusive system is Turing unstable. If we fix $d_1=0.13$, from (16), we can draw Turing bifurcation curve on $p-d_2$ plane (see Figure 15b), and $d_2=d_2^T(5,d_1)$ intersects line $p=p_H$ at $TH (0.8306,0.009)$. Choosing $p=0.8306,d_2=0.0065$ near $TH$, we can illustrate two stable spatially inhomogeneous periodic solutions coexisting when we choose two different initial values (see Figure 16).

6.3. Numerical Simulations for Diffusive System with Two Delays

In this section, we carry out some numerical simulations for diffusive system (4). Fix

$$a=0.23,r=1.1,m=0.31,l=2,$$

which is the same as in (31). Fix $c=0.25$ such that $c<\frac{1}{r(1-a)}$, and choose $p=1.2<p_H$, $d_1=0.3$, and $d_2=0.4$, such that $d_2>d_2^T(1,d_1)$ for all $n\in \mathbb{N}$. From Theorem 9, one can get the unique constant steady state $E^{*}(0.80945,0.11797)$ is locally asymptotically stable.

Now, we illustrate the effect of two delays on the dynamics (4). Following the steps in Appendix C and Appendix D, we can draw all the stability switching curves on $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ plane, and decide the crossing direction, which are shown in Figure A3, Figure A4, Figure A5, Figure A6, Figure A7 in Appendix E. Combining all the stability switching curves together, and zooming in the part of $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)\in [0,5]\times [0,20]$, we get the Hopf bifurcation curves shown in Figure 17a. Consider the bottom left region bounded by left-most curve of ${\mathcal{T}}_0^1$ and the lowest curve of ${\mathcal{T}}_0^2$ (see Figure 17a), which are both part of ${\mathcal{T}}_0$. The crossing directions of the two switching curves (the black line and blue line) are all pointing outside of the region. From Theorem 9, under the assumption $p<p_H$ and $d_2>d_2^T(n,d_1)$, $E^{*}$ is locally asymptotically stable. From Theorem 10, when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross from the left to the right of the stability switching curve ${\mathrm{\tau }}_0^{2\left(1\right)}$ (or cross from the bottom to the top of ${\mathrm{\tau }}_0^{1\left(1\right)}$), the characteristic equation has two more characteristic roots with positive real parts from Lemma A1, $E^{*}$ becomes unstable, and periodic solution occurs due to Hopf bifurcation. Similarly, when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross from the left to the right of stability switching curve ${\mathrm{\tau }}_1^{1\left(1\right)}$ (or from the left to the right of stability switching curve ${\mathrm{\tau }}_2^{1\left(1\right)}$ ), the characteristic equation has two more characteristic roots. Then, $E^{*}$ is still unstable, and system (4) undergoes Hopf bifurcation and periodic solution arises. However, the periodic solution is unstable, as there is unstable manifold. It means that stability switching curves ${\mathrm{\tau }}_0^{2\left(1\right)}$ and ${\mathrm{\tau }}_0^{1\left(1\right)}$ are the boundary of stable region for $E^{*}$. In the stable region in Figure 17a, the predator and the prey coexist, and when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross the stability switching curves ${\mathrm{\tau }}_0^{2\left(1\right)}$ (or ${\mathrm{\tau }}_0^{1\left(1\right)}$), the species coexist in the form of sustained fluctuation. From the previous discussion, if we choose a point (2.31, 0) on ${\mathrm{\tau }}_1-$axis and move up, system (4) undergoes stability switches at $E^{*}$. There are two intersections with the boundary of the stable region: $(2.31,3.25)$, $(2.31,10.41)$ (see Figure 17a). The positive equilibrium $E^{*}$ is unstable when ${\mathrm{\tau }}_2\in [0,2.31)$ and there are periodic solutions; $E^{*}$ becomes locally asymptotically stable and periodic solutions disappear when ${\mathrm{\tau }}_2\in (2.31,10.41)$, and $E^{*}$ loses its stability when ${\mathrm{\tau }}_2>10.41$ and periodic solution appears again. Multiple times stability switching means that the stability of $E^{*}$ can switch with the varying of parameters $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$, and periodic solutions may appear or disappear during the stability switching of $E^{*}$.

Moreover, the two stability switching curves intersect at the point $(2.21407,15.0019)$, which is the double Hopf bifurcation point, denoted by HH. For HH, ${\omega }_1=0.1078$, ${\omega }_2=0.4920$. Using the normal form derivation process given in Supplementary Materials, we have $B_{11}=-0.0512$, $B_{21}=0.0478$, $B_{13}=1.7736$, $B_{23}=-0.0706$, $B_{2100}=-0.7294$, $B_{1011}=-1.5826$, $B_{0021}=-0.2410$, $B_{1110}=-10.8681$. Furthermore, we have the normal form (29) with ${ϵ}_1=-1,{ϵ}_2=-1,b=6.5658,c=14.8998,d=1$, and $d-bc=-96.8306$. According to Chapter 7.5 in [62], case Ib arises, and we have the bifurcation set near HH showing in Figure 17b. In region $D_1$, the positive equilibrium is asymptotically stable. In region $D_2$ or $D_6$, there is a stable periodic solution. When the parameters cross into the region D4, there are two stable homogeneous periodic solutions coexisting in D4.

7. Concluding Remarks

We investigate the dynamics of a diffusive predator–prey model with two delays. First, we discuss the existence of constant coexisting steady states of system (4). It turns out that the strength of cooperation $c=\frac{1}{r(1-a)}$ is a threshold: if $c<\frac{1}{r(1-a)}$, i.e., the cooperation is weak, there is at most one coexisting steady state, and if $c>\frac{1}{r(1-a)}$, i.e., the cooperation is strong, there are at most two coexisting steady states.

To figure out the dynamics of system (4), we first carry out the study on the kinetic predator–prey model (7) with Allee effect in prey and cooperative hunting in predators. Compared with the work of Jang et al. [17], which mainly gives the local bifurcation analysis, we mainly focus on the global dynamics. We start with the kinetic system in the case of weak cooperation. Taking p, the conversion rate, as the bifurcation parameter, we prove that there is a loop of heteroclinic orbits between $E_1$ and $E_a$ formed by the intersection of the stable manifold of $E_a$ and the unstable manifold of $E_1$ when $p=p^{\#}$ ($p^{\#}>1$). It shows that $p^{\#}$ is an important threshold, which distinguishes the globally stability for $E_0$ and the existence of a separatrix curve. To be specific, when $p>p^{\#}$, $E_0$ is globally stable, which means the extinction of both species, and we call the phenomenon overexploitation. When $p<p^{\#}$, there is a separatrix curve ${\mathrm{\Gamma }}_p^s$, which is determined by the stable manifold of $E_a$. High initial predator density will lead to extinction for both species, and conversely low initial predator density will approach either a steady state or a periodic oscillation. When $p<1$, $E_1(1,0)$ is locally stable, which means that the prey achieves its carrying capacity and the predator is extinct. When $1<p<p_H$, the stable state becomes $E^{*}$, which implies the coexistence of both species. When p crosses $p_H$, the stable state switches into a limit cycle. The period of the limit cycle increases with p, and tends to ∞ as $p\to p^{\#}$. The existence of a limit cycle with a large amplitude increases the risk of the extinction for the predator.

In the case of strong cooperation, three different cases are considered. We prove the existence of a loop of heteroclinic orbits between $E_1$ and $E_a$ when $p=p^{\#}$ in the first and second cases and nonexistence of such a loop of heteroclinic orbits in the third case. We also exhibit the existence of loop of heteroclinic orbits among $E_R^{*}$, $E_a$, and $E_1$ formed by the unstable manifold of $E_R^{*}$ and the unstable manifold $E_a$ when $p={\overline{p}}^{\#}$, and a homoclinic cycle when $p=p_{hom}$ by numerical simulations in case 3. When $p>p^{\#}$ in case 1 and case 2 ($p>{\overline{p}}^{\#}$ in case 3), both species are distinct. When $1<p<p^{\#}$ in case 1 and case 2, there is a separatrix curve ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, and there are two stable states coexisting, which are extinction for both species and coexistence or oscillation. When $p<p^{\#}$ (or $p<{\overline{p}}^{\#}$) and $p_{SN}<p<1$, there are two separatrix curves, the stable manifold of $E_a$ and the stable manifold of $E_R^{*}$, separating the first quadrant into three parts. If the initial population is above the stable manifold of $E_a$, both species are extinct; if the initial population is inside the stable manifold of $E_R^{*}$, the stable state can be $E^{*}$ or limit cycle; if the initial population is below the stable manifold of $E_a$ and outside the stable manifold of the stable manifold of $E_R^{*}$, the prey achieves its carrying capacity and the predator is extinct. When $p<p_{SN}$, there is one separatrix curve ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, and there are two stable states coexisting. The species with initial population above the separatrix finally become extinct, however only the prey survive when initial population is below it.

There is a unique equilibrium for weak cooperation when $1<p<\frac{1}{a}$. It is shown that there are at most two stable states coexist for different initial values separated by the stable manifold of $E_a$. There are two interior equilibria $E^{*}$ and $E_R^{*}$ in the case of strong cooperation when $p_{SN}<p<1$. The equilibrium $E_R^{*}$ brings one more separatrix curve than the case of weak cooperation, thus there may be three stable states coexisting.

Second, we consider the diffusive system without delays, and focus on the effect of diffusion on the dynamics. Taking diffusion coefficients $d_1$ and $d_2$ as bifurcation parameters, we give the conditions of existence of Turing instability and Turing–Hopf bifurcation for the system with both weak and strong cooperation. We illustrate complex dynamics of the diffusive system, including the existence of spatially inhomogeneous steady state, coexistence of two spatially inhomogeneous periodic solutions.

Finally, we discuss the joint effect of two delays on the dynamical behavior of the diffusive system. Applying the method of stability switching curves, we find all the stability switching curves at which the characteristic roots are purely imaginary. Combining the geometrical and analytic method, we can decide the crossing direction of the characteristic roots as long as we confirm the positive direction for stability switching curves. Then, we can get the condition of existence of Hopf bifurcation. By searching the intersection of stability switching curves near the stable region, we get the double Hopf bifurcation point. Through the calculation of normal form of the system, we get the corresponding unfolding system and the bifurcation set, from which we can figure out the complete dynamics near the bifurcation point on the $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ parametric plane. We theoretically prove and illustrate that near the bifurcation point, there are the phenomena of the stability of positive equilibrium: stable periodic solutions and the coexistence of two periodic solutions. From double Hopf bifurcation analysis, we can gain an insight into the local dynamics near the double Hopf bifurcation point. However, numerical simulations tells us that the occurrence of complex phenomena near bifurcation point is sensitive to the choice of initial values. When we choose the initial values with high ratio of predator to prey, both species are extinct. Thus, the complex phenomena near the double Hopf bifurcation probably accompany with an extinction steady state, which is consistent with the results in the kinetic system. Therefore, the analysis of global dynamics of kinetic system can help us comprehend these bistable dynamics for the diffusive system.