Dynamics in a Predator–Prey Model with Cooperative Hunting and Allee Effect

Abstract

This paper deals with a diffusive predator–prey model with two delays. First, we consider the local bifurcation and global dynamical behavior of the kinetic system, which is a predator–prey model with cooperative hunting and Allee effect. For the model with weak cooperation, we prove the existence of limit cycle, and a loop of heteroclinic orbits connecting two equilibria at a threshold of conversion rate $p=p^{\#}$, by investigating stable and unstable manifolds of saddles. When $p>p^{\#}$, both species go extinct, and when $p<p^{\#}$, there is a separatrix. The species with initial population above the separatrix finally become extinct, and the species with initial population below it can be coexisting, oscillating sustainably, or surviving of the prey only. In the case with strong cooperation, we exhibit the complex dynamics of system, including limit cycle, loop of heteroclinic orbits among three equilibria, and homoclinic cycle with the aid of theoretical analysis or numerical simulation. There may be three stable states coexisting: extinction state, coexistence or sustained oscillation, and the survival of the prey only, and the attraction basin of each state is obtained in the phase plane. Moreover, we find diffusion may induce Turing instability and Turing–Hopf bifurcation, leaving the system with spatially inhomogeneous distribution of the species, coexistence of two different spatial-temporal oscillations. Finally, we consider Hopf and double Hopf bifurcations of the diffusive system induced by two delays: mature delay of the prey and gestation delay of the predator. Normal form analysis indicates that two spatially homogeneous periodic oscillations may coexist by increasing both delays.

1. Introduction

A predator–prey model can be described by the following equations [1]:

$$\left\{\begin{array}{ccc}\dot{u}\hfill & =\hfill & f\left(u\right)u-G(u,v)v,\hfill \\ \dot{v}\hfill & =\hfill & dG(u,v)v-mv,\hfill \end{array}\right.$$

(1)

where u and v stand for the densities of prey and predator, respectively. $f\left(u\right)$ represents the per capita prey growth rate in the absence of predators, and $G(u,v)$ is the functional response characterizing predation. d describes the rate of biomass conversion from predation, and m is the death rate of predator. Many kinds of functional response have been proposed [2]. Among them, Holling type I, II, and III [3,4,5,6] are discussed widely, which are prey-dependent functional response.

Cooperative behavior within a species is very common in nature, of which cooperative hunting is the most widely distributed form in animals. In mammals, the most famous examples include wolves [7], African wild dogs [8], lions [9], and chimpanzees [10], who cooperate for hunting preys. Cooperative behaviors are also widespread in other living organisms, such as aquatic organisms [11], spiders [12], birds [13], and ants [14], who find, attack, and move their preys together. Due to hunting cooperation, the attack rate of predators increases with predator density, which obviously depends on both prey and predator densities. Recently, a few literatures have paid attention to derive functional response to describe the cooperative hunting [1,15,16]. Cosner et al. [1] proposed a functional response to explore the effects of predator aggregation when predators encounter a cluster of prey. Berec [15] generalized the Holling type-II functional response to a family of functional responses by considering attack rate and handling time of predators varying with predator density to interpret the foraging facilitation among predators. Alves and Hilker [16] considered a special case of the more general functional response proposed by Berec [15]. They added a cooperation term to the attack rate for representing the benefits that hunting cooperation brings to the predator population, which makes the functional response $G(u,v)=(b+cv)u$. Here, $c>0$ describes the strength of predator cooperation in hunting, and $cv$ is referred to as the cooperation term. They found that cooperative hunting can improve persistence of the predator, and it is a form of foraging facilitation which can induce strong Allee effects. After their work, various types of biological factors (such as Allee effect, fear effect, and so on) and spatial diffusion are incorporated into continuous-time predator–prey models with cooperative hunting [17,18,19,20,21,22]. The impact of hunting cooperation in discrete-time predator–prey models have also been investigated [23,24,25,26]. Pal et al. [23] observed that hunting cooperation can modify the period-doubling route to chaos by reverse period-halving bifurcations and makes the system stable. Chow et al. [26] derived a sufficient condition for which both populations can coexist.

The most well-known example of $f\left(u\right)$ is the logistic form $f\left(u\right)=r\left(1-\frac{u}{K}\right)$, where r is the intrinsic growth rate and K denotes the environmental carrying capacity. Then, in the absence of predator, the prey population is governed by $\dot{u}=ru\left(1-\frac{u}{K}\right)$. We can conclude from the equation that the species may increase in size when the density is low. However, for many species, low population density may induce many problems, such as mate difficulties, inbreeding, and predator avoidance of defense. It turns out that the growth rate of the low density population is not always positive, and it may be negative when the density of population is less than the minimum number a for the survival of the population, which is called the Allee threshold. The phenomenon is known as the Allee effect, which was first observed by Allee [27], and has been observed on various organisms, such as vertebrates, invertebrates, and plants. Such a population can be described by $\dot{u}=ru\left(1-\frac{u}{K}\right)(u-a)$. The Allee effect may cause the extinction of a low-density population. Because of increasing fragmentation of habitats and invasions of exotic species, the Allee effect has aroused increasing attention (see, for example, in [28,29,30] and the references cited therein).

Jang et al. [17] considered both the cooperation in the predator and Allee effects in the prey, and proposed the following predator–prey model:

$$\left\{\begin{array}{ccc}\dot{u}\hfill & =\hfill & r_{1}u\left[1-\frac{u}{K_1}\right]\left[u-a_1\right]-\left[b_1+c_{1}v\right]uv,\hfill \\ \dot{v}\hfill & =\hfill & p_1[b_1+c_{1}v]uv-m_{1}v,\hfill \end{array}\right.$$

(2)

where $r_1$ is the per capita intrinsic growth rate of the prey, $K_1$ is the environmental carrying capacity for the prey, $a_1$ ($a_1<K_1$) is the Allee threshold of the prey population, $b_1$ is the attack rate per predator and prey, $c_1$ measures the degree of cooperation of predator, $p_1$ is the prey conversion to predator, and $m_1$ is the per capita death rate of predator. They discussed the existence and stability of the equilibria and the occurrence of Hopf bifurcation. They also devised a best strategy for culling the predator by maximizing the prey population and minimizing the predators along with the costs associated with the control.

In fact, the preys and predators distribute inhomogeneously in different locations, and the spatial diffusion plays an important part in the process of population evolution. Turing instability and Turing–Hopf bifurcation induced by diffusion have been widely investigated recently (see in [31,32,33,34]).

It has been widely accepted that time delays have very complex effect on the dynamics of a system, for example, some delays can destroy the stability of equilibria and induce various oscillations and periodic solutions (see, for example, in [35,36,37,38] and references cited therein). There are time delays in almost every process of population interaction, so it is more realistic to introduce time delays when we model the interaction of predator and prey. For example, after consuming the prey, the corresponding biomass cannot be added to the predator population immediately. Thus, the reproduction of predator after predating the prey will be mediated by some discrete time lag (say ${\mathrm{\tau }}_2$) required for gestation of predator, which can be regarded as gestation delay. Therefore, the growth rate of predator depends on the number of individuals present at time $t-{\mathrm{\tau }}_2$. There is an extensive literature about the studies of the dynamics of predator–prey models with the effect of gestation delay of the predator [35,36,37,38]. There is a time lag (say ${\mathrm{\tau }}_1$) in the growth to maturity of the prey, and such a maturation delay of the prey has also been considered [39,40,41,42,43]. However, to the best of our knowledge, there is very little literatures on delayed predator–prey system with Allee effect [44].

Taking into account the diffusion and delays in system (2), we consider the following diffusive system with two delays:

$$\left\{\begin{array}{c}\frac{\partial u(x,t)}{\partial t}=d_1\Delta u(x,t)+r_{1}u(x,t)[1-\frac{u(x,t-{\mathrm{\tau }}_1)}{K_1}](u(x,t)-a_1)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-(b_1+c_{1}v(x,t))u(x,t)v(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ \frac{\partial v(x,t)}{\partial t}=d_2\Delta v(x,t)+p_1\left[b_1+c_{1}v(x,t-{\mathrm{\tau }}_2)\right]u(x,t-{\mathrm{\tau }}_2)v(x,t)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-mv(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ {\partial }_{v}u(x,t)=0, {\partial }_{v}v(x,t)=0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x=0,l\pi , t>0\hfill \\ u(x,t)={\mathsf{\Phi }}_1(x,t)\ge 0, v(x,t)={\mathsf{\Phi }}_2(x,t)\ge 0,\hspace{1em}\hspace{1em}\hspace{1em}x\in [0,l\pi ], t\in [-\mathrm{\tau },0].\hfill \end{array}\right.$$

(3)

Here, ${\mathrm{\tau }}_1$ is the time delay due to the maturation of the prey, ${\mathrm{\tau }}_2$ is time delay due to gestation of the predator, and we define $\mathrm{\tau }=\max \{{\mathrm{\tau }}_1,{\mathrm{\tau }}_2\}$. $d_1,d_2>0$ are the diffusion coefficients characterizing the rates of the spatial dispersion of the prey and predator population, respectively.

Systems with multiple delays have attracted much attention [45,46,47,48,49,50,51]. Generally, delay may induce Hopf bifurcation, and if Hopf bifurcation curves intersect, double Hopf bifurcation may arise. To figure out the effect of delay on the dynamics of systems, Hopf bifurcation and double Hopf bifurcation induced by delay have been investigated [52,53,54,55,56]. However, in most of literatures we mentioned above, the system is reduced into a system with one delay. In fact, systems with multiple delays conform to reality better than one single delay. Recently, the research on the dynamics and bifurcation analysis of system with two simultaneously varying delays are of great interest to scholars. In [49,57], double Hopf bifurcation induced by two delays is studied by different methods. Through the analysis of double Hopf bifurcation, we can classify the topological structures of various bifurcating solutions. By the classification, the dynamics in the neighborhood of the double Hopf bifurcation point in system can be obtained completely.

For the convenience of discussion and reducing the number of parameters, we nondimensionalize the model by letting

$$\widehat{u}=\frac{1}{K_1}u, \widehat{v}=b_{1}v.$$

Dropping the hats for simplicity of notations, we obtain the simplified dimensionless system

$$\left\{\begin{array}{c}\frac{\partial u(x,t)}{\partial t}=d_1\Delta u(x,t)+ru(x,t)[1-u(x,t-{\mathrm{\tau }}_1)](u(x,t)-a)\hfill \\ \hspace{1em}\hspace{1em} -(1+cv(x,t\left)\right)u(x,t)v(x,t),\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x\in (0,l\pi ), t>0\hfill \\ \frac{\partial v(x,t)}{\partial t}=d_2\Delta v(x,t)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em} +mv(x,t)\left[pu(x,t-{\mathrm{\tau }}_2)(1+cv(x,t-{\mathrm{\tau }}_2))-1\right], x\in (0,l\pi ), t>0\hfill \\ {\partial }_{v}u(x,t)=0, {\partial }_{v}v(x,t)=0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}x=0,l\pi , t>0\hfill \\ u(x,t)={\mathsf{\Phi }}_1(x,t)\ge 0, v(x,t)={\mathsf{\Phi }}_2(x,t)\ge 0,\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} x\in [0,l\pi ], t\in [-\mathrm{\tau },0],\hfill \end{array}\right.$$

(4)

where $r=K_1{r}_1, a=\frac{a_1}{K_1}, c=\frac{c_1}{b_1^2}, \mathrm{and} p=\frac{b_1{p}_1{K}_1}{m}$. Recalling the definition of Allee effect, one always has $0<a<1$. In fact, for fixed $K_1$, $r_1$, $a_1$, $b_1$, and m, we have c and p directly proportional to $c_1$ and $p_1$, respectively. Thus, we refer to c and p as the degree of cooperative hunting of the predator and the conversion rate from the prey to the predator, respectively, in the context. By the explanation above, we make the following assumption always:

$$\left(H\right) r, a, c, m, p \mathrm{a}\mathrm{n}\mathrm{d} c \mathrm{a}\mathrm{r}\mathrm{e} \mathrm{a}\mathrm{l}\mathrm{l} \mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e},\mathrm{a}\mathrm{n}\mathrm{d} 0<a<1.$$

The Allee effect can drive the extinction of low-density species, some of which are preyed by predators that engage in cooperative hunting. This paper investigate the dynamics of system (4) with cooperative hunting and Allee effect from the point of view bifurcation analysis, which can provide useful insights into the intersection of the predator and prey, and help us figure out how to prevent such a prey population from extinction. We find out that the number of constant steady states of (4) is highly dependent on the strength of cooperation in predator c. When $c<\frac{1}{r(1-a)}$, there is one or no positive steady states, while when $c>\frac{1}{r(1-a)}$, there is no, one or two positive steady states. We say the cooperation in the former case is weak, and that in the latter is strong. First of all, we figure out the global dynamical behavior of the kinetic system through the analysis of local bifurcation and the stable and unstable manifolds, taking p as a bifurcation parameter. The existence of some connecting orbits is obtained, thus, a partition of the phase space is given, i.e., the attraction basin of each stability state is obtained. For the model with weak cooperation, we give the condition of the stability of the positive steady state and the occurrence of Hopf bifurcation. Through detailed analysis on the stable and unstable manifolds of saddles, we prove theoretically there is a loop of heteroclinic orbits when $p=p^{\#}$. When $p>p^{\#}$, the extinction steady state $E_0(0,0)$ is globally asymptotically stable. When $p<p^{\#}$, there is a separatrix, which is the stable manifold of $E_a$. The orbits above the separatrix converge to $E_0$, and the orbits below it may converge to $E_1$, $E^{*}$ or a limit cycle. Thus, there are three types of bistability: bistability between two boundary equilibria, between a boundary equilibrium and a interior equilibrium, and between a boundary equilibrium and a limit cycle. For the model with strong cooperation, it turns out there are many complex phenomena, including limit cycle, homoclinic cycle, and a loop of heteroclinic orbits between two or among three equilibria. The model may display tristable dynamics, including extinction state, coexistence or sustained oscillation, and the survival of the prey only.

Second, we discuss the diffusive system (4) with no delays. Taking diffusive coefficients as parameters, we give the condition of Turing instability and Turing–Hopf bifurcation in both weak and strong cooperation cases. We discuss the effect of diffusion on the dynamics, and illustrate complex spatio-temporal dynamical behavior, including the existence of spatially inhomogeneous steady state, coexistence of two spatially inhomogeneous periodic solutions.

Third, taking two time delays as bifurcation parameters, we study the effect of two simultaneous varying delays on the dynamics of system (4). After some detailed analysis on the characteristic equation with three exponential terms, the stability switching curves are obtained, on which there is a pair of purely imaginary eigenvalues. An intersection of the stability switching curves may be a double Hopf bifurcation point. We derive the normal form on the center manifold near the double Hopf bifurcation point, and give the unfoldings near the critical points, which indicate that two spatially homogeneous periodic oscillations may coexist by increasing both delays.

The scheme of this paper is arranged as follows. In Section 2, we investigate the existence of constant steady states for system (4). In Section 3, we analyze the local bifurcation and the global dynamics of the kinetic system (7) with weak cooperation and strong cooperation, respectively. In Section 4, we consider the effect of diffusion on system (4) through investigating Turing instability and Turing–Hopf bifurcation. In Section 5, we consider the diffusive system with two delays, and Hopf bifurcation and double Hopf bifurcation induced by two delays are analyzed on the center manifold via normal form approach. In Section 6, we carry out some numerical simulations to support our theoretical analysis and illustrate complex dynamical phenomenon.

2. The Existence of Constant Steady States

First, we consider the existence of steady states of model (4). As (4) is two-dimensional, it is hard to consider non-constant steady states. Thus, we only discuss the existence of constant steady states.

System (4) has three boundary equilibria: (i) $E_0(0,0)$, which means the extinction of both the species; (ii) $E_a(a,0)$, which is induced by Allee effect; and (iii) $E_1(1,0)$, which means the extinction of the predator and the survival of the prey, achieving its carrying capacity.

Now, we discuss the existence of interior equilibria. Obviously, any interior equilibrium has components $u>0$ and $v>0$ satisfying

$$\begin{array}{cc}\hfill r(1-u)(u-a)& =(1+cv)v,\hfill \\ \hfill pu(1+cv)& =1,\hfill \end{array}$$

(5)

i.e.,

$$\begin{array}{cc}\hfill r(1-u)(u-a)& =\frac{1-pu}{c{p}^2{u}^2},\hfill \\ \hfill v& =\frac{1-pu}{cpu}.\hfill \end{array}$$

(6)

In fact, $r(1-u)(u-a)=(1+cv)v$ is an ellipse, sitting in the first and forth quadrant, and intersecting the horizontal axis at a and 1. $pu(1+cv)=1$ intersects the horizontal axis at $\frac{1}{p}$, and is decreasing and concave in $(0,\frac{1}{p})$. Denote the u-nullcline $r(1-u)(u-a)=(1+cv)v$ as $v=f\left(u\right)$, and the v-nullcline $pu(1+cv)=1$ as $v=g\left(u\right)$. The existence of interior equilibria may have the following different cases.

Proposition 1.

(i) When $p\ge \frac{1}{a}$, system (4) has no interior equilibria.

(ii) 

When $1<p<\frac{1}{a}$, system (4) has a unique positive constant equilibrium $E^{*}(u^{*},v^{*})$ with $a<u^{*}<\frac{1}{p}$.

(iii) 

Suppose that $p<1$. If $c<\frac{1}{r(1-a)}$, system (4) has no interior equilibria. If $c>\frac{1}{r(1-a)}$, there exists a $p_{SN}<1$, such that system (4) has

(a) 

two interior equilibria $E^{*}$ and $E_R^{*}$ when $p_{SN}<p<1$;

(b) 

one interior equilibrium $E_R^{**}$ when $p=p_{SN}$; and

(c) 

no interior equilibria when $p<p_{SN}$.

Proof. 

In fact, when $a<\frac{1}{p}<1$, two nullclines have two intersections: $E^{*}$ in the first quadrant and $E_R^{*}$ in the forth quadrant. Thus, there is a unique interior equilibrium $E^{*}(u^{*},v^{*})$ with $u^{*}$ and $v^{*}$ satisfying (5). When $\frac{1}{p}$ decreases to a, $E^{*}$ collides with $E_a$, and $E_R^{*}$ is still in the forth quadrant. When $\frac{1}{p}<a$, $E^{*}$ moves into the forth quadrant, leaving no interior equilibria.

When $\frac{1}{p}$ increases to 1, v-nullcline and u-nullcline intersect at $E_1(1,0)$, with the slopes of tangents at $E_1(1,0)$ being $-\frac{1}{c}$ and $-r(1-a)$, respectively. If $-\frac{1}{c}<-r(1-a)$, $E^{*}$ collides with $E_1$ when $p=1$, $E_R^{*}$ is in the forth quadrant, and there is no interior equilibria (see Figure 1a). Moreover, when $p<1$, there is no interior equilibria. If $-\frac{1}{c}>-r(1-a)$, $E_R^{*}$ (sitting in the forth quadrant when $1<p<\frac{1}{a}$) collides with $E_1$ when $p=1$, and there is a unique interior equilibrium $E^{*}$ in the first quadrant (see Figure 1b). When $p<1$ and close to 1, $E_R^{*}$ moves into the first quadrant, and there are two interior equilibria $E^{*}$ and $E_R^{*}$. If $p<1$ goes on to decrease, there exists a $p_{SN}$, such that the u-nullcline $v=f\left(u\right)$ and v-nullcline $v=g\left(u\right)$ tangent at $E_R^{**}$, where $E^{*}$ and $E_R^{*}$ collide, and there is a unique interior equilibrium $E_R^{**}$. When $p<p_{SN}$, there is no interior equilibria. □

Figure 1. When $p=1$, (a) there is no interior equilibrium if $c<\frac{1}{r(1-a)}$, and (b) there is one interior equilibrium if $c>\frac{1}{r(1-a)}$.

Remark 1.

In the absence of cooperative hunting within the predator, i.e., $c=0$, system (4) always has a unique interior equilibrium $E_0^{*}=(\frac{1}{p},r(1-\frac{1}{p})(\frac{1}{p}-a))$. The predator and prey coexist if and only if $1<p<\frac{1}{a}$, and the predator is distinct if $p<1$. However, from proposition 1, the predator and prey still coexist when $p_{SN}<p<1$ with strong cooperation, which means that strong cooperation is beneficial for the survival of predator.

Proposition 1 tells us that the strength of cooperation $c=\frac{1}{r(1-a)}$ is a threshold for the system (4). To be specific, if $c<\frac{1}{r(1-a)}$, system (4) has at most one coexisting equilibrium, while if $c>\frac{1}{r(1-a)}$, system (4) may have two coexisting equilibria. Thus, in the following discussion, when $c<\frac{1}{r(1-a)}$, we say the cooperation among predators is weak; when $c>\frac{1}{r(1-a)}$, the cooperation is strong.

3. Bifurcation and Global Dynamics of Kinetic System

Before the discussion of the stability of equilibria of system (4), we should first figure out the stability of the equilibria of the kinetic system

$$\left\{\begin{array}{c}\dot{u}=ru(1-u)(u-a)-(1+cv)uv,\hfill \\ \dot{v}=mv\left[pu\left(1+cv\right)-1\right].\hfill \end{array}\right.$$

(7)

We investigate the local bifurcation and global dynamics when the cooperation is weak and strong, i.e., when $c<\frac{1}{r(1-a)}$ and $c>\frac{1}{r(1-a)}$.

3.1. Bifurcation and Global Dynamics of Kinetic System with Weak Cooperative Hunting

From proposition 1, if $c<\frac{1}{r(1-a)}$, system (4) has a unique interior equilibrium. In this section, taking p as bifurcation parameter, we first investigate the Hopf bifurcation near the unique interior equilibrium. Through studying the stable manifold and unstable manifold of saddles, we prove the existence of loop of heteroclinic orbits, and get the global dynamics of system (7).

3.1.1. Stability of All Equilibria and Hopf Bifurcation at $E^{*}$

The Jacobian matrices of the function on the right-hand of system (7) around $E_0$, $E_a$, and $E_1$ are

$$J_{E_0}=\left(\begin{array}{cc}-ra& 0\\ 0& -m\end{array}\right),$$

$$J_{E_a}=\left(\begin{array}{cc}ra(1-a)& -a\\ 0& m(pa-1)\end{array}\right),$$

$$J_{E_1}=\left(\begin{array}{cc}-r(1-a)& -1\\ 0& m(p-1)\end{array}\right),$$

respectively, from which we can easily get the local stability of the boundary equilibria.

Lemma 1.

For system (7),

 (i)

$E_0(0,0)$ is a stable node;

 (ii)

$E_a(a,0)$ is an unstable node if $p>\frac{1}{a}$, and it is a saddle if $p<\frac{1}{a}$; and

 (iii)

$E_1(1,0)$ is a stable node if $p<1$, and it is a saddle if $p>1$.

For the interior equilibrium $E^{*}(u^{*},v^{*})$, the corresponding Jacobian matrix is

$$J_{E^{*}}=\left(\begin{array}{cc}r{u}^{*}(1+a-2u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ mp{v}^{*}(1+c{v}^{*})& mpc{u}^{*}{v}^{*}\end{array}\right),$$

(8)

and thus

$$\begin{array}{c}\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+mpc{u}^{*}{v}^{*}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}),\hfill \\ \det J_{E^{*}}=mp{u}^{*}{v}^{*}\left[rc{u}^{*}(1+a-2u^{*})+(1+c{v}^{*})(1+2c{v}^{*})\right].\hfill \end{array}$$

In fact, u-nullcline $v=f\left(u\right)$ and v-nullcline $v=g\left(u\right)$ intersect at $E^{*}$, where the slope of $v=f\left(u\right)$ is larger than that of $v=g\left(u\right)$, i.e., $\frac{1+c{v}^{*}}{-c{u}^{*}}<\frac{r(1+a-2u^{*})}{1+2c{v}^{*}}$. Thus, $\det J_{E^{*}}>0$. Thus $E^{*}$ may be node or focus, and its stability depends on the trace

$$\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}).$$

(9)

Thus, we have the following conclusions on the stability of $E^{*}$.

Theorem 1.

If $c<\frac{1}{r(1-a)}$, then there exists a unique $p_H\in (1,\frac{2}{a+1})$ such that the unique interior equilibrium $E^{*}$ is locally asymptotically stable when $1<p<p_H$, and unstable when $p_H<p<\frac{1}{a}$. Moreover, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

Proof. 

Solving for $\mathrm{tr}{J}_{E^{*}}=0$, when $\frac{a+1}{2}<\frac{1}{p}<1$, we obtain a unique $u_H^{*}\in (\frac{a+1}{2},1)$, such that $\mathrm{tr}{J}_{E^{*}}>0$ on $(0,u_H^{*})$, and $\mathrm{tr}{J}_{E^{*}}<0$ on $(u_H^{*},1)$ (see Figure 2). Moreover, it is easy to verify that $\frac{\partial u^{*}}{\partial p}<0$; thus, corresponding to $u_H^{*}$, we obtain a unique $p_H$ ($1<p_H<\frac{2}{a+1}$), such that $\mathrm{tr}{J}_{E^{*}}>0$ when $p\in (p_H,\frac{1}{a})$, and $\mathrm{tr}{J}_{E^{*}}<0$ when $p\in (1,p_H)$.

Figure 2. The blue curve represents $y=r{u}^{*}(2u^{*}-a-1)$, and the black line represents $y=m(1-p{u}^{*})$. When $u=u_H^{*}$, $\mathrm{tr}{J}_{E^{*}}=0$.

Now, we verify the transversality condition. Let $\mu =\alpha \left(p\right)\pm i\omega \left(p\right)$ be the roots of ${\mu }^2-\mathrm{tr}{J}_{E^{*}}\mu +\det J_{E^{*}}=0$ when p is near $p_H$. We have ${\alpha }^{\prime }\left(p\right)=\frac{1}{2}\frac{\mathrm{d}\mathrm{tr}{J}_{E^{*}}}{\mathrm{d}p}=\frac{1}{2}[-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)-2r{u}^{*}{u}^{{*}^{\prime }}\left(p\right)$$-m{u}^{*}-mp{u}^{{*}^{\prime }}\left(p\right)]$, where $u^{*}\left(p\right)$ is a function of p determined by (6). Taking the derivative of both sides of (6) with respect to p, we have

$$-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)=\frac{(-u-p{u}^{{*}^{\prime }}\left(p\right))[c{p}^2{u}^{*2}+2cp{u}^{*}(1-p{u}^{*})]}{c^2{p}^4{u}^{*4}},$$

thus

$$-u-p{u}^{{*}^{\prime }}\left(p\right){\mid }_{p=p_H}=\frac{-r{u}^{{*}^{\prime }}\left(p\right)(2u^{*}-a-1)c^2{p}^4{u}^{*4}}{c{p}^2{u}^{*2}+2cp{u}^{*}(1-p{u}^{*})}{\mid }_{p=p_H}>0,$$

as $\frac{a+1}{2}<u_H^{*}<\frac{1}{p}$ and $u^{{*}^{\prime }}\left(p\right)<0$. Thus, ${\alpha }^{\prime }\left(p\right){\mid }_{p=p_H}>0$. □

Remark 2.

When there is no cooperative hunting in predator, i.e., $c=0$, the v-nullcline is $u=\frac{1}{p}$, which is vertical. $\mathrm{tr}{J}_{E^{*}}{{|}_{c=0}=-\frac{r}{p}(\frac{2}{p}-a-1),\det J_{E^{*}}|}_{c=0}=m{v}^{*}.$ The system undergoes a Hopf bifurcation near $E^{*}$ when $\frac{1}{p}=\frac{a+1}{2}$, which is the top of the u-nullcline $v=r(1-u)(u-a)$. In fact, it has been discussed widely that the stability of positive equilibrium can be stated graphically by the u-nullcline when the v-nullcline is vertical: $(u^{*},v^{*})$ is unstable if the v-nullclines intersect to the left of a local maximum of the u-nullcline, and stable if they intersect to the right [29,58,59,60], and thus, Hopf bifurcation occurs at the “top of the hump” of u-nullcline. However, for system (7) in this paper, the v-nullcline is not vertical, and we can verify that the intersection of the u-nullcline and the v-nullcline corresponding to the Hopf bifurcation point $p=p_H$ is on the right of the “top of the hump" of u-nullcline. In fact, the “top of the hump” of u-nullcline $v=f\left(u\right)$ achieves at $u=\frac{a+1}{2}$, at which the corresponding value of p is denoted by $p_{\mathrm{top}}$. When $p=p_{\mathrm{top}}$, $E^{*}$ can be also proved to be unstable. Comparing the results in systems with and without cooperation, we can conclude that cooperative hunting is more likely to bring instability into $E^{*}$.

In order to determine the direction of Hopf bifurcation and the stability of the Hopf bifurcating periodic solution, we need to calculate the normal form near the Hopf bifurcation point. Through direct calculation following the steps in [61], we can get the following truncated normal form:

$$\begin{array}{c}\dot{r}={\alpha }^{\prime }\left(p_H\right)pr+a\left(p_H\right)r^3+O\left(r\right|p-p_H{|}^2,r^3|p-p_H|,r^5),\hfill \\ \dot{\theta }=\omega \left(p_H\right)+{\omega }^{\prime }\left(p_H\right)(p-p_H)+c\left(p_H\right)r^2+O\left(\right|p-p_H{|}^2,r^2|p-p_H|,r^4).\hfill \end{array}$$

Recalling that ${\alpha }^{\prime }\left(p_H\right)>0$, the direction of Hopf bifurcation and stability of Hopf bifurcating periodic solution are determined by the first Lyapunov coefficient $a\left(p_H\right)$, and we have the following conclusion.

Theorem 2.

If $c<\frac{1}{r(1-a)}$, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

 (i)

If $a\left(p_H\right)<0$, the bifurcating periodic solution is orbitally asymptotically stable, and it is bifurcating from $E^{*}$ as p increases and passes $p_H$.

 (ii)

If $a\left(p_H\right)>0$, the bifurcating periodic solution is unstable, and it is bifurcating from $E^{*}$ as p decreases and passes $p_H$.

3.1.2. The Global Dynamics of Kinetic System with Weak Cooperative Hunting

From the previous discussion, we have known the stability of all equilibria when $c<\frac{1}{r(1-a)}$. Motivated by the work of [29,30], we investigate the global dynamics of system (7). To this end, we should first study the properties of stable manifold of $E_a$ and unstable manifold of $E_1$, denoted by ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$, respectively. For the region bounded by ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$, we prove the existence or nonexistence of periodic orbits. We leave them in Appendix A, and only list main conclusions on the existence of a loop of heteroclinic orbit and global dynamics in the following.

From the monotonicity of S and U given in Proposition A1, we have the following conclusion.

Proposition 2.

If $c<\frac{1}{r(1-a)}$, then there exists a unique $p^{\#}\in (1,\frac{1}{a})$, such that ${\mathrm{\Gamma }}_{p^{\#}}^s={\mathrm{\Gamma }}_{p^{\#}}^u$, forming a heteroclinic orbit from $E_1$ to $E_a$.

In fact, there is another heteroclinic orbit from $E_a$ to $E_1$, which is formed by the unstable manifold of $E_a$ and the stable manifold of $E_1$ on the $u-$axis. Thus, there is a loop of heteroclinic orbits from $E_1$ to $E_a$, and then back to $E_1$.

Theorem 3.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

If $p\ge \frac{1}{a}$, $E_0(0,0)$ is globally asymptotically stable (see Figure 3a).

Figure 3. Phase portrait of system (7) for $c<\frac{1}{r(1-a)}$ when (a) $p>\frac{1}{a}$; (b) $p<\frac{1}{a}$ and close to $\frac{1}{a}$; (c) $p>1$ and close to 1; (d) and $p<1$.

 (ii)

There is an ${\epsilon }_1>0$ such that if $\frac{1}{a}-{\epsilon }_1<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 3b).

 (iii)

There is an ${\epsilon }_2>0$ such that if $1<p<1+{\epsilon }_2$, then ${\mathrm{\Gamma }}_p^u$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E^{*}$ (see Figure 3c).

 (iv)

If $p<1$, the orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E_1$ (see Figure 3d).

We leave the proof in Appendix A.2.

Theorem 3 provides a global description of the dynamical behaviour of system (7) for $p\in {\mathbb{R}}^{+}\backslash (1+{\epsilon }_2,\frac{1}{a}-{\epsilon }_1)\stackrel{}{=}{\mathbb{R}}^{+}\backslash I$. When $p\in I$, if every periodic orbit of system (7) is orbitally stable, thus there can be at most one such orbit. From Proposition A5, we have the following conclusion.

Theorem 4.

Suppose that $c<\frac{1}{r(1-a)}$, the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable. Then, $p_H<p^{\#}$.

 (i)

If $1<p<p_H$, ${\mathrm{\Gamma }}_p^u$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to $E^{*}$.

 (ii)

If $p_H<p<p^{\#}$, $E^{*}$ is a repellor, and there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s$. The orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to the limit cycle. (see Figure 4a,b).

Figure 4. Phase portrait of (7) for $c<\frac{1}{r(1-a)}$ when (a) $p_H<p<p^{\#}$ and close to $p_H$; (b) $p_H<p<p^{\#}$ and close to $p^{\#}$; (c) $p=p^{\#}$; (d) $p^{\#}<p<\frac{1}{a}$.

 (iii)

If $p=p^{\#}$, ${\mathrm{\Gamma }}_p^s={\mathrm{\Gamma }}_p^u$, there are two heteroclinic orbits forming a loop of heteroclinic orbits from $E_1$ to $E_a$ and back to $E_1$ (see Figure 4c). The orbits through any point exterior to the loop converge to $E_0$, and the orbits through any point interior to the cycle converge to the loop.

 (iv)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 4d).

Remark 3.

We leave the proof in Appendix A.3.

From the former conclusion on the global dynamics of model (7) with weak cooperation, we find that no matter how we choose the value of p, both of the two species are extinct if the ratio of predator to prey is high.

Now, we discuss the population behavior when the ratio of predator to prey is low. When $p<1$, because of the low conversion rate, the predator is extinct, while the prey reach the capacity of the environment. When $1<p<p_H$, the predator and prey coexist, and tend to a constant value. When $p_H<p<p^{\#}$, the predator and prey coexist, but oscillate sustainably. The amplitude of oscillation increases as p increases, and the minimum of the predator population is close to zero when p is close to $p^{\#}$, which will increase of risk on the extinction of predator. When the value of p is too high, high growth rate of prey leads to over hunting, and finally, both of the two species are extinct.

Remark 4.

Similar to the case in the model with Allee effect, there is a unique positive equilibrium in the model without Allee effect, and the varying p makes $E^{*}$ unstable and induces Hopf bifurcation. However, there is an additional saddle point $(a,0)$ in the model with Allee effect, and the stable manifold of the saddle point separates the positive quadrant into two positive invariant regions. If the initial populations are above the stable manifold of $(a,0)$, extinction happens for both species in the model with Allee effect, while coexistence happens for the model without Allee effect. This extinction is mediated by too high predator density that drives the prey density below the Allee threshold and both species become extinct.

3.2. Bifurcation and Global Dynamics of Kinetic System with Strong Cooperative Hunting

By strong cooperation we mean $c>\frac{1}{r(1-a)}$. In this section, we first analyze the stability of interior equilibrium, then we prove the existence of Hopf bifurcation and give the condition of the existence and nonexistence of loop of heteroclinic orbits. We also exhibit the complex dynamics of model (7), such as limit cycle, loop of heteroclinic orbits, and homoclinic cycle.

3.2.1. Stability and Hopf Bifurcation of Interior Equilibria

Now, we consider the stability of interior equilibria. For $E^{*}$ or $E_R^{*}$, the corresponding Jacobian matrix is

$$J=\left(\begin{array}{cc}ru(1+a-2u)& -2cuv-u\\ mpv(1+cv)& mpcuv\end{array}\right),$$

(10)

and thus

$$\begin{array}{c}\mathrm{tr}J=-ru(2u-a-1)+mpcuv=-ru(2u-a-1)+m(1-pu),\hfill \\ \det J=mpuv\left[rcu(1+a-2u)+(1+cv)(1+2cv)\right]\hfill \\ =\frac{mv}{u}\left[rcp{u}^3(1+a-2u)+\frac{2}{p}-u\right].\hfill \end{array}$$

If $\det J=0$, then $\frac{r(1+a-2u)}{1+2cv}=\frac{1+cv}{-cu}$, which means that $v=f\left(u\right)$ is tangent to $v=g\left(u\right)$. In fact, the point of tangency is $E_R^{**}$, thus $\det J_{E_R^{**}}=0$. It is clear that $\det J>0$ for $u<u_R^{**}$, and $\det J<0$ for $u>u_R^{**}$ (see Figure 5). The component $u^{*}$ of $E^{*}$ is always satisfying $u^{*}<u_R^{**}$, and the component $u_R^{*}$ of $E_R^{*}$ is satisfying $u_R^{*}>u_R^{**}$ (see Figure 1). It follows that for $E^{*}$, $\det J_{E^{*}}>0$, while for $E_R^{*}$, $\det J_{E_R^{*}}<0$. Thus, $E_R^{*}$ exists when $p_{SN}<p<1$, and it is a saddle. $E^{*}(u^{*},v^{*})$ exists when $p_{SN}<p<\frac{1}{a}$, and $\det J_{E^{*}}>0$. Thus, $E^{*}$ may be node or focus, and its stability depends on the trace

$$\mathrm{tr}{J}_{E^{*}}=-r{u}^{*}(2u^{*}-a-1)+m(1-p{u}^{*}).$$

(11)

Figure 5. The red curve represents the figure of $y=rcp{u}^3(2u-a-1)$, and the green straight line represents $y=2/p-u$.

Similar as the proof in Theorem 1, solving for $\mathrm{tr}{J}_{E^{*}}=0$, when $\frac{a+1}{2}<\frac{1}{p}<\frac{1}{p_{SN}}$, there is a unique $u_H^{*}\in (\frac{a+1}{2},\frac{1}{p_{SN}})$ such that $\mathrm{tr}{J}_{E^{*}}>0$ on $(0,u_H^{*})$, and $\mathrm{tr}{J}_{E^{*}}<0$ on $(u_H^{*},\frac{1}{p_{SN}})$ (see Figure 2). Recalling that $\frac{\partial u^{*}}{\partial p}<0$, corresponding to $u_H^{*}$, we obtain a unique $p_H$ ($p_{SN}<p_H<\frac{2}{a+1}$), such that $\mathrm{tr}{J}_{E^{*}}>0$ when $p\in (p_H,\frac{1}{a})$, and $\mathrm{tr}{J}_{E^{*}}<0$ when $p\in (p_{SN},p_H)$.

Let $\mu =\alpha \left(p\right)\pm i\omega \left(p\right)$ be the roots of ${\mu }^2-\mathrm{tr}{J}_{E^{*}}\mu +\det J_{E^{*}}=0$ when p is near $p_H$. We can prove ${\alpha }^{\prime }\left(p\right){\mid }_{p=p_H}>0$, in a similar way as in Theorem 1. Therefore, we can get the following theorem.

Theorem 5.

When $c>\frac{1}{r(1-a)}$, there exists a unique $p_H\in (p_{SN},\frac{2}{a+1})$ such that $E^{*}$ is locally asymptotically stable if $p_{SN}<p<p_H$, and unstable if $p_H<p<\frac{1}{a}$. Moreover, system (7) undergoes a Hopf bifurcation at $E^{*}$ when $p=p_H$.

3.2.2. The Global Dynamics of Kinetic System with Strong Cooperative Hunting

When $c>\frac{1}{r(1-a)}$, system (7) may have one or two interior equilibria, which brings a few complications to the dynamics of model (7). There are three different cases of global dynamics when c takes different sizes. In this section, we exhibit the dynamics combining theoretical analysis and numerical simulations.

First, we consider the properties of the stable manifold of $E_a$, unstable manifold of $E_1$, as well as stable and unstable manifold of $E_R^{*}$, which are in Appendix B. Then, we can determine the global dynamics of model (7) when p is chosen in the following ranges. As in Theorem 3, we can prove the following conclusion.

Theorem 6.

Suppose that $c>\frac{1}{r(1-a)}$.

 (i)

If $p\ge \frac{1}{a}$, $E_0(0,0)$ is globally asymptotically stable (see Figure 6a).

Figure 6. If $c>\frac{1}{r(1-a)}$, phase portrait of (7) when (a) $p\ge \frac{1}{a}$; (b) $p<\frac{1}{a}$ and close to $\frac{1}{a}$; and (c) $p<p_{SN}$.

 (ii)

If $p<\frac{1}{a}$ and near $\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (see Figure 6b).

 (iii)

If $p<p_{SN}$, the orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_1$ (see Figure 6c).

The remaining question is how are the dynamics of system (7) for p chosen in the rest ranges, i.e., $p_{SN}<p<\frac{1}{a}$. As larger c means less steep of the v-nullcline, the critical points (for example, $p_H$, $p^{\#}$) may appear in different ranges, and there are the following three different cases.

Case 1 Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H>1$. From Theorem A1, there is a loop of heteroclinic orbits when $p=p^{\#}$. We have the following conclusion.

Theorem 7.

Suppose that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H>1$. Assume that the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable, then $p_H<p^{\#}$.

 (i)

If $p_{SN}<p<1$, the upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ and exterior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E_1$ (see Figure 7a).

Figure 7. Phase portrait of system (7) for case 1 when (a) $p>p_{SN}$ and close to $p_{SN}$, (b) $p>p_H$ and close to $p_H$, (c) $p_H<p<p^{\#}$ and close to $p^{\#}$, and (d) $p=p^{\#}$.

 (ii)

If $1<p<p_H$, ${\mathrm{\Gamma }}_p^u\left(E_1\right)$ connects $E_1$ to $E^{*}$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E^{*}$.

 (iii)

When $p_H<p<p^{\#}$, $E^{*}$ is a repellor, and there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s\left(E_a\right)$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to the limit cycle (see Figure 7b,c).

 (iv)

When $p=p^{\#}$, ${\mathrm{\Gamma }}_{p^{\#}}^s\left(E_a\right)={\mathrm{\Gamma }}_{p^{\#}}^u\left(E_1\right)$, and there are two heteroclinic orbits forming a loop of heteroclinic orbits between $E_1$ and $E_a$ (see Figure 7d). The orbits through any point exterior to the cycle converge to $E_0$, and the orbits through any point interior to the cycle converge to the cycle.

 (v)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (phase portrait is similar as Figure 6b).

Proof. 

From Proposition A6, $S_a\left(p\right)$ is decreasing with p, combining with $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, then ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ enters $E_a$ from the region $\left\{\right(u,v):v>f(u\left)\right\}$ for all $p<1$. As $E^{*}$ is stable when $p<p_H$, then $E^{*}$ is stable for all $p_{SN}<p<1$. If there is a periodic orbit below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, it must encircle $E^{*}$, and it is unstable from inside, which contradicts with our assumption of the orbital stability of periodic orbit. It means that there is no periodic orbit under ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ for all $p<1$. From Proposition A9, $E^{*}$ is the $\omega -$limit set of ${\mathrm{\Omega }}_{22}$, and ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ must be above ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$. The upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$ (see Figure 7a). Thus, the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The proof of (ii)–(v) is similar as that of Theorem 4. □

Case 2 Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H<1$. From Theorem A1, there is a loop of heteroclinic orbits when $p=p^{\#}$.

Theorem 8.

Choose suitable c such that $c>\frac{1}{r(1-a)}$, $U_1\left(1^{+}\right)<S_a\left(1^{+}\right)$, and $p_H<1$. Assume that the first Lyapunov coefficient $a\left(p_H\right)<0$, and every periodic orbit of system (7) is orbitally stable, then $p_H<p^{\#}$.

 (i)

If $p_{SN}<p<p_H$, the upward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E^{*}$, and the downward ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ connects $E_R^{*}$ to $E_1$. The orbits through any point above ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ converge to $E_0$, and the orbits through any point inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E^{*}$. The orbits through any point below ${\mathrm{\Gamma }}_p^s\left(E_a\right)$ and exterior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to $E_1$ (phase portrait is similar as Figure 7a).

 (ii)

When $p_H<p<1$, there is a unique limit cycle inside the stable manifold ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ of $E_R^{*}$ (see Figure 8a). The orbits through any point interior to ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ converge to the limit cycle.

Figure 8. Phase portrait of system (7) for case 2 when (a) $p_H<p<1$ and close to $p_H$; (b) $p_H<p<p^{\#}$ and close to $p^{\#}$; (c) $p=p^{\#}$.

 (iii)

When $1<p<p^{\#}$, there is a unique limit cycle under ${\mathrm{\Gamma }}_p^s$. The orbits through any point below ${\mathrm{\Gamma }}_p^s$ converge to the limit cycle (see Figure 8b).

 (iv)

When $p=p^{\#}$, ${\mathrm{\Gamma }}_{p^{\#}}^s\left(E_a\right)={\mathrm{\Gamma }}_{p^{\#}}^u\left(E_1\right)$, there are two heteroclinic orbits forming a loop of heteroclinic orbits from $E_1$ to $E_a$ and back to $E_1$ (see Figure 8c). The orbits through any point exterior to the cycle converge to $E_0$, and the orbits through any point interior to the cycle converge to the cycle.

 (v)

If $p^{\#}<p<\frac{1}{a}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and the extinction equilibrium $E_0(0,0)$ is globally asymptotically stable (phase portrait is similar as Figure 6b).

Proof. 

We can prove (i) similar as in Theorem 7. (ii) As $a\left(p_H\right)<0$ and ${\alpha }^{\prime }\left(p_H\right)>0$, there is a stable periodic orbit bifurcating from Hopf bifurcation when $p>p_H$. As in the proof of Theorem 7, we can prove that ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ must be above ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$. From Proposition A9, the unique limit cycle is the $\omega -$limit set of ${\mathrm{\Omega }}_{22}$. The proof of (iii)–(v) is similar as that of Theorem 4. □

Case 3 Choose c such that $c>\frac{1}{r(1-a)}$ and $U_1\left(1^{+}\right)>S_a\left(1^{+}\right)$. From Theorem A1, there is no heteroclinic orbits from $E_1$ to $E_a$. It is easy to prove that when $1<p<\frac{1}{a}$, $E_0$ is globally asymptotically stable (see Figure 9e).

Figure 9. Phase portrait of system (7) for case 3 when (a) $p>p_H$ and near $p_H$, there is a limit cycle inside the stable manifold of $E_R^{*}$; (b) $p<p_{hom}$ and near $p_{hom}$, there is a limit cycle inside the stable manifold of $E_R^{*}$; (c) $p=p_{hom}$, the upward unstable manifold and the left stable manifold of $E_R^{*}$ collide, which forms a homoclinic cycle; (d) the upward unstable manifold of $E_R^{*}$, the unstable manifold of $E_a$ on the $u-$axis, and the downward unstable manifold of $E_R^{*}$ forms a loop of heteroclinic orbits among $E_R^{*}$, $E_a$, and $E_1$ when $p={\overline{p}}^{\#}$; and (e) $1<p<\frac{1}{a}$.

Remark 5.

In this case, the location of ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, ${\mathrm{\Gamma }}_p^s\left(E_R^{*}\right)$ and ${\mathrm{\Gamma }}_p^u\left(E_R^{*}\right)$ are very complicated. We can observe the following dynamics by numerical simulations.

 (i)

Loop of heteroclinic orbits. The downward branch of the unstable manifold of $E_R^{*}$ connects $E_R^{*}$ to $E_1$, and the upward connects $E_R^{*}$ to $E_a$, which collides with the the stable manifold of $E_a$. The upward and downward unstable manifold of $E_R^{*}$, together with the unstable manifold of $E_a$ on the $u-$ axis, forms a loop of heteroclinic orbits among $E_R^{*}$, $E_a$ and $E_1$ when $p={\overline{p}}^{\#}$ (see Figure 9d).

 (ii)

Homoclinic cycle. The upward unstable manifold and the left stable manifold of $E_R^{*}$ collide, which forms a homoclinic cycle. Denote the parameter p as $p=p_{hom}$ (see Figure 9c).

 (iii)

Limit cycle induced by Hopf bifurcation (see Figure 9a,b).

Remark 6.

There are two positive equilibria $E^{*}$ and $E_R^{*}$ in both of the model with or without Allee effect, where $E_R^{*}$ is always a saddle. In the model without Allee effect, the stable manifold of $E_R^{*}$ separates the positive quadrant into two positive invariant regions. Two species coexist if the initial populations are above the stable manifold. However, there is an additional stable manifold of saddle point $(a,0)$ in the model with Allee effect, and two stable manifolds of the two saddle points separate the positive quadrant into three positive invariant regions. Two species only coexist between the two stable manifolds, and both species go extinct if the initial populations is above the stable manifold of $(a,0)$. This means that the region of coexistence of two species is reduced due to the Allee effect.

4. Diffusion-Driven Turing Instability and Turing–Hopf Bifurcation

In the previous section, we have discussed both local bifurcation and global dynamics of kinetic model (7). In this section, we consider the effect of the diffusion on the stability of the interior constant steady states; thus, we set ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$ in system (4). If an equilibrium is linearly stable in the absence of diffusion, and it becomes unstable in the presence of diffusion, we call such an instability Turing instability. As $E_R^{*}$ is always a saddle if it exists, thus Turing instability can only happen near $E^{*}$ in both cases: weak cooperation and strong cooperation. We first investigate the existence of Turing instability, then we consider Turing–Hopf bifurcation near $E^{*}$.

For Neumann boundary condition, we define the real-valued Sobolev space

$$X=\{{(u,v)}^T\in H^2(0,l\pi )\times H^2(0,l\pi ), \frac{\partial u}{\partial x}=\frac{\partial v}{\partial x}=0,\mathrm{at} x=0,l\pi \}.$$

(12)

The linearization of system (4) with ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$ at the constant steady state $E^{*}(u^{*},v^{*})$ is given by

$$\left(\begin{array}{c}\frac{\partial u}{\partial t}\\ \frac{\partial v}{\partial t}\end{array}\right)=D\left(\begin{array}{c}\Delta u\\ \Delta v\end{array}\right)+A\left(\begin{array}{c}u\\ v\end{array}\right)\stackrel{∆}{=}L\left(\begin{array}{c}u\\ v\end{array}\right),$$

(13)

where

$$D=\left(\begin{array}{cc}{d}_1& 0\\ 0& d_2\end{array}\right), A=\left(\begin{array}{cc}r{u}^{*}(1+a-2u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ mp(1+c{v}^{*})v^{*}& mpc{u}^{*}{v}^{*}\end{array}\right).$$

It is well known that the eigenvalues of $D\Delta$ on X are $-d_1\frac{n^2}{l^2}$ and $-d_2\frac{n^2}{l^2}$, $k\in {\mathbb{N}}_0=\{0,1,2,...\}$. From straightforward calculation, we obtain the characteristic equations

$${\Delta }_n={\lambda }^2-T_n\lambda +J_n=0, n=0,1,2,...,$$

(14)

where

$$\begin{array}{c}{T}_n=-\left(d_1\frac{n^2}{l^2}+d_2\frac{n^2}{l^2}+r{u}^{*}(2u^{*}-a-1)-mpc{u}^{*}{v}^{*}\right)\hfill \\ =-d_1\frac{n^2}{l^2}-d_2\frac{n^2}{l^2}+\mathrm{tr}{J}_{E^{*}},\hfill \\ J_n=d_1{d}_2\frac{n^4}{l^4}-r{u}^{*}(1+a-2u^{*})d_2\frac{n^2}{l^2}-mpc{u}^{*}{v}^{*}{d}_1\frac{n^2}{l^2}+\det J_{E^{*}}.\hfill \end{array}$$

(15)

From the previous section, in the absence of diffusion, i.e., $d_1=d_2=0$, $E^{*}$ is asymptotically stable if $1<p<p_H$ in the case of $c<\frac{1}{r(1-a)}$ (if $p_{SN}<p<p_H$ in the case of $c>\frac{1}{r(1-a)}$). If there exists an $n\in \mathbb{N}$, such that ${\Delta }_n=0$ has roots with positive real part when $p<p_H$, Turing instability occurs. As $\mathrm{tr}{J}_{E^{*}}<0$ when $p<p_H$, and $\det J_{E^{*}}>0$ always satisfies, from (15), we have $T_n<0$ for any n when $p<p_H$. Therefore, when $p<p_H$, the signs of the real parts of roots of (14) are determined by the signs of $J_n$.

In fact, the curve of $J_n=0$ is a hyperbola on $d_1-d_2$ plane, whose horizontal and vertical asymptotes are $d_2=\frac{mpc{u}^{*}{v}^{*}{l}^2}{n^2}$ and $d_1=-\frac{r{u}^{*}(2u^{*}-1-a)l^2}{n^2}$. We only need to consider the right branch of the hyperbola, which intersects with the $d_1-$axis at $d_1=\frac{\det J_{E^{*}}{l}^2}{mpc{u}^{*}{v}^{*}{n}^2}$. Thus, with respect to n, both the horizontal asymptote of the right branch of the hyperbola and the intersection with the $d_1-$axis are decreasing.

Denote

$$d_2^T(n,d_1)\stackrel{∆}{=}\frac{mpc{u}^{*}{v}^{*}{d}_1\frac{n^2}{l^2}-\det J_{E^{*}}}{d_1\frac{n^4}{l^4}+r{u}^{*}(2u^{*}-1-a)\frac{n^2}{l^2}}.$$

(16)

Noting that $u^{*}>\frac{a+1}{2}$ when $p\le p_H$, we have $J_n<0$ if $d_2<d_2^T(n,d_1)$, and $J_n>0$ if $d_2>d_2^T(n,d_1)$.

Theorem 9.

If $c<\frac{1}{r(1-a)}$, suppose $1<p<p_H$ (If $c>\frac{1}{r(1-a)}$, suppose $p_{SN}<p<p_H$). The constant steady state $E^{*}$ is locally asymptotically stable if $d_2>d_2^T(n,d_1)$ for all $n\in \mathbb{N}$, while it is Turing unstable if $d_2<d_2^T(n,d_1)$ for some $n\in \mathbb{N}$ satisfying $n>l\sqrt{\frac{\det J_{E^{*}}}{mpc{u}^{*}{v}^{*}{d}_1}}$.

On the $d_1-d_2$ plane, we call the boundary curve of stable region of steady state $E^{*}$ the Turing bifurcation curve $l^T$, which is formed by a sequence of curve segments $l_n^T$$(n=1,2,...)$

$$d_2=d_2^T(n,d_1), \mathrm{for} d_{1,n}<d_1\le d_{1,n-1},$$

(17)

where $d_{1,n}$ is the intersection of $d_2=d_2^T(n+1,d_1)$ and $d_2=d_2^T(n,d_1)$, and $d_{1,0}$ can be infinite.

When $p=p_H$, $T_n<0$ for all $n\in \mathbb{N}$, similar as the analysis in the case of $p<p_H$, we can also get a Turing bifurcation curve, and we have the following conclusion.

Remark 7.

Assume, in the $p-d_2$ plane, $d_2=d_2^T(n,d_1)$ and $p=p_H$ intersect at a point $TH$, then the characteristic equation has a pair of purely imaginary roots and a zero root at $TH$. According to the general theory of [62], a Turing–Hopf bifurcation may appear. However, rigorous derivation for the normal form is impossible due to the implicit form of $E^{*}$ in this paper. We shall give some numerical illustrations to show the dynamics of the system near the Turing–Hopf bifurcation point.

5. The Dynamics of Diffusive System with Two Delays

In the previous section, we have found the conditions of stability and Turing instability of constant steady state $E^{*}$ of system (4) when ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$. We know that if $p<p_H$ and $d_2>d_2^T(n,d_1)$ for all $n\in \mathbb{N}$, the steady state $E^{*}$ is locally asymptotically stable. In this section, we investigate the diffusive system (4) under the assumption $p<p_H$ and $d_2>d_2^T(n,d_1)$, and focus on the effect of two delays on the dynamics of the diffusive system near $E^{*}$.

5.1. Hopf and Double Hopf Bifurcation Induced by Two Delays

In this subsection, we investigate the existence of Hopf bifurcation induced by two delays by the method of stability switching curves given in [63], and give the condition of the existence of double Hopf bifurcation.

The linearization of system (4) at the steady state $E^{*}(u^{*},v^{*})$ is given by

$$\frac{\partial }{\partial t}\left(\begin{array}{c}u(x,t)\hfill \\ v(x,t)\hfill \end{array}\right)=(D\Delta +A)\left(\begin{array}{c}u(x,t)\hfill \\ v(x,t)\hfill \end{array}\right)+B\left(\begin{array}{c}u(x,t-{\mathrm{\tau }}_1)\hfill \\ v(x,t-{\mathrm{\tau }}_1)\hfill \end{array}\right)+C\left(\begin{array}{c}u(x,t-{\mathrm{\tau }}_2)\hfill \\ v(x,t-{\mathrm{\tau }}_2)\hfill \end{array}\right),$$

(18)

where

$$\begin{array}{c}A=\left(\begin{array}{cc}r{u}^{*}(1-u^{*})& -2c{u}^{*}{v}^{*}-u^{*}\\ 0& 0\end{array}\right), B=\left(\begin{array}{cc}-r{u}^{*}(u^{*}-a)& 0\\ 0& 0\end{array}\right),\hfill \\ C=\left(\begin{array}{cc}0& 0\\ mp(1+c{v}^{*})v^{*}& mpc{u}^{*}{v}^{*}\end{array}\right),D=\mathrm{diag}(d_1,d_2).\hfill \end{array}$$

(19)

$u(x,t)$ and $v(x,t)$ satisfy the homogeneous Neumann boundary condition.

From Wu [64], the corresponding characteristic equation of (18) is

$$\det \left(\lambda I-M_n-A-B{e}^{-\lambda {\mathrm{\tau }}_1}-C{e}^{-\lambda {\mathrm{\tau }}_2}\right)=0,$$

(20)

where I is a $2\times 2$ identity matrix, $M_n=-n^2/l^2\mathrm{diag}(d_1,d_2)$, $n\in {\mathbb{N}}_0=\{0,1,2,...\}$. Equation (20) can be written in the following form:

$$D_n(\lambda ;{\mathrm{\tau }}_1,{\mathrm{\tau }}_2)=P_{0,n}\left(\lambda \right)+P_{1,n}\left(\lambda \right)e^{-\lambda {\mathrm{\tau }}_1}+P_{2,n}\left(\lambda \right)e^{-\lambda {\mathrm{\tau }}_2}+P_{3,n}\left(\lambda \right)e^{-\lambda ({\mathrm{\tau }}_1+{\mathrm{\tau }}_2)}=0,$$

(21)

where

$$\begin{array}{c}{P}_{0,n}\left(\lambda \right)=(\lambda +d_1\frac{n^2}{l^2}-a_{11})(\lambda +d_2\frac{n^2}{l^2}),\hfill \\ P_{1,n}\left(\lambda \right)=-b_{11}(\lambda +d_2\frac{n^2}{l^2}),\hfill \\ P_{2,n}\left(\lambda \right)=-c_{22}(\lambda +d_1\frac{n^2}{l^2}-a_{11})-a_{12}{c}_{21},\hfill \\ P_{3,n}\left(\lambda \right)=b_{11}{c}_{22}.\hfill \end{array}$$

(22)

The characteristic equation with the form of (21) has been investigated by Lin and Wang [63]. They derived an explicit expression for the stability switching curves, on which there is a pair of purely imaginary roots for (21). Moreover, they gave a criterion to determine the crossing directions, i.e., on which side of the stability switching curve there are two more characteristic roots with positive real parts. Using this method, we can find all the stability switching curves in the $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ plane and determine the crossing directions. We leave the details in Appendix C and Appendix D.

Moreover, we have the following Hopf bifurcation theorem with two parameters.

Theorem 10.

For each $j\in \{1,2,\cdots ,N\}$, ${\mathcal{T}}_n^j$, defined by (A9), is a Hopf bifurcation curve in the following sense: for any $p\in {\mathcal{T}}_n^j$ and for any smooth curve Γ intersecting with ${\mathcal{T}}_n^j$ transversely at p, we define the tangent of Γ at p by $\overrightarrow{l}$. If $\frac{\partial \mathrm{Re}\lambda }{\partial \overrightarrow{l}}{\mid }_p\ne 0$, and the other eigenvalues of (21) at p have non-zero real parts, then system (4) undergoes a Hopf bifurcation at p when parameters $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross ${\mathcal{T}}_n^j$ at p along Γ.

The theorem can be proved by a similar method in [49].

Remark 8.

Suppose that ${\mathcal{T}}_{k_1}^{j_1}$ and ${\mathcal{T}}_{k_2}^{j_2}$ intersect at a point $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$, with the corresponding values of ω being ${\omega }_{j_1,k_1}\in {\mathrm{\Omega }}_{j_1,k_1}$ and ${\omega }_{j_2,k_2}\in {\mathrm{\Omega }}_{j_2,k_2}$. Then, there are two pairs of purely imaginary roots of (21) at the intersection, which are $\pm i{\omega }_{j_1,k_1}$ and $\pm i{\omega }_{j_2,k_2}$, denoted by $\pm i{\omega }_1$ and $\pm i{\omega }_2$ for convenience. Thus, system (4) may undergo double Hopf bifurcations at the intersection of two stability switching curves near the constant steady state $E^{*}$.

5.2. Normal Form on the Center Manifold for Double Hopf Bifurcation

In this subsection, applying the normal form method of partial functional differential equations [65], we derive normal form of double Hopf bifurcation taking two delays as bifurcation parameters. Then, we can classify the topological structures of various bifurcating solutions, and get the dynamics in the neighborhood of the double Hopf bifurcation point in system (4).

For the real-valued Sobolev space X defined in (12), define the complexification space of the real-valued Hilbert space X by

$$X_{\mathbb{C}}:=X\oplus iX=\{U_1+i{U}_2:U_1,U_2\in X\},$$

with the general complex-value $L^2$ inner product

$$⟨U,V⟩={\int }_0^{l\pi }({\overline{u}}_1{v}_1+{\overline{u}}_2{v}_2)dx, \mathrm{f}\mathrm{o}\mathrm{r} U={(u_1,u_2)}^T,V={(v_1,v_2)}^T\in X_{\mathbb{C}}.$$

Let $\mathcal{C}:=C([-1,0],X_{\mathbb{C}})$ denote the phase space with the sup norm, and write $u^t\in \mathcal{C}$ for $u^t\left(\theta \right)=u(t+\theta )$, $-1\le \theta \le 0$.

Without loss of generality, we assume ${\mathrm{\tau }}_1<{\mathrm{\tau }}_2$ in this section. Denote the double Hopf bifurcation point by $({\mathrm{\tau }}_1^{*},{\mathrm{\tau }}_2^{*})$. Let $\overline{u}(x,t)=u(x,{\mathrm{\tau }}_{2}t)-u^{*},\overline{v}(x,t)=v(x,{\mathrm{\tau }}_{2}t)-v^{*}$, and set ${\sigma }_1={\mathrm{\tau }}_1-{\mathrm{\tau }}_1^{*}$ and ${\sigma }_2={\mathrm{\tau }}_2-{\mathrm{\tau }}_2^{*}$ as two bifurcation parameters. Drop the bars, and denote $U\left(t\right)={(u\left(t\right),v\left(t\right))}^T$, then system (4) can be written as

$$\frac{dU\left(t\right)}{dt}=D({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\Delta U\left(t\right)+L({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\left(U^t\right)+F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,U^t),$$

(23)

where

$$\begin{array}{c}D({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)=({\mathrm{\tau }}_2^{*}+{\sigma }_2)D={\mathrm{\tau }}_2^{*}D+{\sigma }_{2}D,\hfill \\ L({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2)\left(\varphi \right)=({\mathrm{\tau }}_2^{*}+{\sigma }_2)\left[A\varphi \left(0\right)+B\varphi (-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})+C\varphi (-1))\right],\hfill \\ F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,\varphi )=({\mathrm{\tau }}_2^{*}+{\sigma }_2){(F_1,F_2)}^T,\hfill \end{array}$$

with $A, B,C$ and $D$ being defined in (19), and

$$\begin{array}{c}{F}_1=r(1-u^{*}){\varphi }_1^2\left(0\right)-(1+2c{v}^{*}){\varphi }_1\left(0\right){\varphi }_2\left(0\right)+(ra-2r{u}^{*}){\varphi }_1\left(0\right){\varphi }_1(-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})\hfill \\ -c{u}^{*}{\varphi }_2^2\left(0\right)-r{\varphi }_1^2\left(0\right){\varphi }_1(-\frac{{\mathrm{\tau }}_1^{*}+{\sigma }_1}{{\mathrm{\tau }}_2^{*}+{\sigma }_2})-c{\varphi }_1\left(0\right){\varphi }_2^2\left(0\right),\hfill \end{array}$$

$$\begin{array}{c}{F}_2=mp(1+c{v}^{*}){\varphi }_2\left(0\right){\varphi }_1(-1)+mpc[u^{*}{\varphi }_2\left(0\right)+v^{*}{\varphi }_1(-1)+{\varphi }_2\left(0\right){\varphi }_1(-1)]{\varphi }_2(-1).\hfill \end{array}$$

Consider the linearized system of (23)

$$\frac{dU\left(t\right)}{dt}={\mathrm{\tau }}_2^{*}D\Delta U\left(t\right)+{\mathrm{\tau }}_2^{*}(A{U}^t\left(0\right)+B{U}^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*})+C{U}^t(-1))\stackrel{}{=}{D}_0\Delta U\left(t\right)+L_0\left(U^t\right).$$

(24)

We know that the normalized eigenfunctions of $D\Delta$ on X corresponding to the eigenvalues $-d_1\frac{n^2}{l^2}$ and $-d_2\frac{n^2}{l^2}$( $n\in {\mathbb{N}}_0=\{0,1,2,\cdots \}$) are

$${\beta }_n^1\left(x\right)={\gamma }_n\left(x\right){(1,0)}^T \mathrm{and} {\beta }_n^2\left(x\right)={\gamma }_n\left(x\right){(0,1)}^T,$$

respectively, where ${\gamma }_n\left(x\right)=\frac{cos\frac{n}{l}x}{\Vert cos\frac{n}{l}{x\Vert }_{L^2}}$. Define ${\mathcal{B}}_n:=\mathrm{span}\left\{⟨v(·),{\beta }_n^j⟩{\beta }_n^j | v\in \mathcal{C},j=1,2\right\}$, satisfying $L\left({\mathcal{B}}_n\right)({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)\subset \mathrm{span}\{{\beta }_n^1,{\beta }_n^2\}$. Denote $⟨v(·),{\beta }_n⟩={\left(⟨v(·),{\beta }_n^1⟩,⟨v(·),{\beta }_n^2⟩\right)}^T.$

Write system (23) as

$$\frac{dU\left(t\right)}{dt}=D_0\Delta U\left(t\right)+L_0\left(U^t\right)+\tilde{F}(\sigma ,U^t),$$

(25)

where

$$\begin{array}{c}\hfill \tilde{F}(\sigma ,U^t)={\sigma }_2(D\Delta U^t\left(0\right)+A{U}^t\left(0\right)+B{U}^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*})+C{U}^t(-1))\\ \hfill +({\mathrm{\tau }}_2^{*}+{\sigma }_2)B(U^t(-({\mathrm{\tau }}_1^{*}+{\sigma }_1)/({\mathrm{\tau }}_2^{*}+{\sigma }_2))-U^t(-{\mathrm{\tau }}_1^{*}/{\mathrm{\tau }}_2^{*}))+F({\mathrm{\tau }}_1^{*}+{\sigma }_1,{\mathrm{\tau }}_2^{*}+{\sigma }_2,U^t).\end{array}$$

System (25) can be rewritten as an abstract ordinary differential equation on $\mathcal{C}$ [65]

$$\frac{d}{dt}{U}^t=\mathcal{A}{U}^t+X_0\tilde{F}(\sigma ,U^t),$$

(26)

where $\mathcal{A}$ is the infinitesimal generator of the $C_0$-semigroup of solution maps of the linear equation (23), defined by

$$\mathcal{A}\phi =\dot{\phi }+X_0[D_0\Delta \phi \left(0\right)+L_0\left(\phi \right)-\dot{\phi }\left(0\right)],$$

(27)

with $\mathrm{dom}\left(\mathcal{A}\right)=\{\phi \in \mathcal{C}:\dot{\phi }\in \mathcal{C},\phi \left(0\right)\in \mathrm{dom}(\Delta )\}$, and $X_0$ is given by $X_0\left(\theta \right)=0$ for $\theta \in [-1,0)$ and $X_0\left(0\right)=I$. Clearly, $\mathcal{A}:{\mathcal{C}}_0^1\cap \mathcal{C}\to \mathcal{C}.$

Then, on ${\mathcal{B}}_n$, the linear equation

$$\frac{d}{dt}U\left(t\right)=D_0\Delta U\left(t\right)+L_0\left(U^t\right)$$

is equivalent to the retarded functional differential equation on ${\mathbb{C}}^2$:

$$\dot{z}\left(t\right)=-\frac{n^2}{l^2}{D}_{0}z\left(t\right)+L_0{z}^t.$$

(28)

By the Riesz representation theorem, there exists a matrix whose components are bounded variation functions ${\eta }_k\in BV([-1,0],{\mathbb{R}}^{2\times 2})$ such that

$$-\frac{n_k^2}{l^2}{D}_0\phi \left(0\right)+L_0\left(\phi \right)={\int }_{-1}^{0}d{\eta }_k\left(\theta \right)\phi \left(\theta \right),\phi \in \mathcal{C}.$$

Let $A_k$ ($k=1,2$) denote the infinitesimal generator of the semigroup generated by (28), and $A_k^{*}$ denotes the formal adjoint of $A_k$ under the bilinear form

$${(\alpha ,\beta )}_k=\alpha \left(0\right)\beta \left(0\right)-{\int }_{-1}^0{\int }_0^{\theta }\alpha (\xi -\theta )d{\eta }_k\left(\theta \right)\beta \left(\xi \right)d\xi .$$

The calculations of normal form are very long, so we leave them in supplementary materials. Based on the derivation in the Supplementary Materials, the normal form truncated to the third order on the center manifold for double Hopf bifurcation is obtained. Making the polar coordinate transformation, then we obtain the following system corresponding to the truncated normal form

$$\begin{array}{c}{\dot{\rho }}_1=r_1({\nu }_1+r_1^2+b{r}_2^2),\hfill \\ {\dot{\rho }}_2=r_2({\nu }_2+c{r}_1^2+d{r}_2^2),\hfill \end{array}$$

(29)

where

$${\nu }_1={ϵ}_1(\mathrm{Re}{B}_{11}{\sigma }_1+\mathrm{Re}{B}_{21}{\sigma }_2),{\nu }_2={ϵ}_1(\mathrm{Re}{B}_{13}{\sigma }_1+\mathrm{Re}{B}_{23}{\sigma }_2),$$

$$b=\frac{{ϵ}_1{ϵ}_2\mathrm{Re}{B}_{1011}}{\mathrm{Re}{B}_{0021}},c=\frac{\mathrm{Re}{B}_{1110}}{\mathrm{Re}{B}_{2100}},d={ϵ}_1{ϵ}_2,$$

and $B_{11},B_{21},B_{13},B_{23},B_{1011},B_{0021},B_{1110}$ and $B_{2100}$ are the coefficients of the normal form obtained in Supplementary Materials. From Chapter 7.5 in [62], there are twelve distinct kinds of unfoldings for (29).

6. Numerical Simulations

In this section, we carry out some numerical simulations for the kinetic system (7), diffusive system without delays, and the diffusive system (4) with two delays, respectively. These numerical simulations confirm and illustrate our main theoretical results.

6.1. Numerical Simulations for Kinetic System

In this subsection, we carry out some numerical simulations for system (7). We fix the parameters as

$$a=0.23,r=1.1,m=0.31.$$

(30)

6.1.1. Numerical Simulations for Kinetic System with Weak Cooperative Hunting

For system (7), choose $c=0.25$ such that $c<\frac{1}{r(1-a)}$. To show the complex dynamics of system (7), different values of p are chosen, listed in Table 1 and Table 2. The corresponding phase portraits for different values of p have been illustrated in Figure 3 and Figure 4, which is drawn by pplane8 [66].

Table 1. Values of parameter p chosen in Figure 3 and Figure 4.

Figure	Figure 3a	Figure 3b	Figure 3c	Figure 3d
p	5.1	3.2	1.2	0.9
Figure	Figure 4a	Figure 4b	Figure 4c	Figure 4d
p	1.55	1.645	1.6491	1.95

Table 2. Values of parameter p chosen for showing case 1 in Figure 7 and Figure 8.

Figure	Figure 7a	Figure 7b	Figure 7c	Figure 7d
p	0.96	1.105	1.2	1.2068
Figure	Figure 8a	Figure 8b	Figure 8c
p	0.965	1.052	1.05665

Using the method in [67], we can get Hopf bifurcation point $p_H=1.5432$, and the first Lyapunov coefficient is $-1.1211$, which means that the bifurcating periodic solution is asymptotically stable, and it is bifurcating from $E^{*}$ as p increases past $p_H$ from Theorem 2. We can draw the bifurcation diagram in Figure 10. If $p<p_H=1.5432$, $E^{*}$ is asymptotically stable (the black curve in Figure 10a). The increase of p induces instability to $E^{*}$. When $p>1.5432$, $E^{*}$ becomes unstable (the red curve in Figure 10a), and a unique limit cycle appears (marked by LPC). The period of limit cycle is increasing as p increases from $p_H$, which tends to infinite as $p\to 1.6491$ (see Figure 10b). When $p=p^{\#}=1.6491$, the limit cycle coincides with a loop of heteroclinic orbits. When $p>p^{\#}=1.6491$, the loop of heteroclinic orbits breaks and the limit cycle disappears. The corresponding phase portrait has been drawn in Figure 4, and the results are consistent with Theorem 4. Moreover, the amplitude of oscillation increases as p increases, and the minimum of the predator population is close to zero when p is close to $p^{\#}=1.6491$, which will increase the risk on the extinction of predator.

Figure 10. The bifurcation diagram with $a=0.23,m=0.31,r=1.1$, and $c=0.25$, and the conversion rate p as the bifurcation parameter. (a) The black curve is the stable steady state and the red curve is the unstable steady state. The blue curves denote the $v-$amplitude of the limit cycle, which begin at the Hopf bifurcation point $p=p_H=1.5432$. (b) The period of the limit cycles.

6.1.2. Numerical Simulations for Kinetic System with Strong Cooperative Hunting

In Section 3.2, we discuss three different cases of dynamics when c is chosen as different values. To show case 1, we fix $c=3$ and vary p as a bifurcation parameter. We get $p_H=1.1024$, and the first Lyapunov coefficient is $-2.3280$. Choosing different values of p listed in Table 3, we have drawn the corresponding phase portraits for different values of p in Figure 7.

Table 3. Values of parameter p chosen for showing case 3 in Figure 9.

Figure	Figure 9a	Figure 9b	Figure 9c	Figure 9d	Figure 9e
p	0.834	0.88	0.8919	0.91564	1.1

To show case 2, choosing $c=5$ and varying p, we can get $p_H=0.9608$, and the first Lyapunov coefficient is $-2.9553$. Choosing p as listed in Table 2, the corresponding phase portraits have been illustrated in Figure 8.

To show case 3, choosing $c=8$ and varying p, we can get $p_{SN}=0.7665$, $p_H=0.8306$, and the first Lyapunov coefficient is $-3.732$. We can draw the bifurcation diagram in Figure 11. When $p<0.7665$ (i.e., on the left of LP), there is no interior equilibria. When $p>0.7665$ (i.e., on the right of LP), there are two equilibria $E_{*}$ (the black curve) and $E_R^{*}$ (the blue dotted curve). If $0.7665<p<0.8306$, $E^{*}$ is asymptotically stable (the blue curve). The increase of p induces instability to $E^{*}$. When $p>0.8306$, $E^{*}$ becomes unstable (the red curve), and a unique limit cycle appears (marked by LPC in Figure 11a). When p increasing from $p_H$, the period of limit cycle is increasing, and it tends to infinite as $p\to 0.8919$ (see Figure 11b). Moreover, the amplitude of oscillation increases as p increases, and the amplitude line touches the curve of $v-$value of $E_R^{*}$. It means that there is a cycle goes through $E_R^{*}$, which forms a homoclinic cycle. The limit cycle coincides with a homoclinic cycle when $p=p_{hom}=0.8919$. When $p>0.8919$, the homoclinic cycle breaks, and the limit cycle disappears. Choosing p as listed in Table 3, the corresponding phase portraits have been illustrated in Figure 9.

Figure 11. The bifurcation diagram with $a=0.23,m=0.31,r=1.1$, and $c=8$, and the conversion rate p as the bifurcation parameter. (a) The black curve is the stable steady state $E^{*}$, the red curve is the unstable steady state $E^{*}$, and the dotted blue curve is the unstable steady state $E_R^{*}$. The blue curves denote the $v-$amplitude of the limit cycle, which begin at the Hopf bifurcation point $p=p_H=0.8306$. (b) The period of the limit cycles.

6.2. Numerical Simulations for Diffusive System without Delays

In this section, we carry out some numerical simulations for diffusive system (4) with ${\mathrm{\tau }}_1={\mathrm{\tau }}_2=0$. We fix the parameters as

$$a=0.23,r=1.1,m=0.31,l=2.$$

(31)

To illustrate the dynamics of (4) with no delays in the case of weak cooperation, we choose $c=0.25$ such that $c<\frac{1}{r(1-a)}$. Let $p=1.4<p_H=1.5432$, from the previous section, $E^{*}(0.6882,0.1516)$ is stable when $d_1=d_2=0$. On $d_1-d_2$ plane, we can draw curves determined by (16) for $n\in \mathbb{N}$, and we only illustrate four curves for $n=2,3,4,5$. Correspondingly, from (17), we can get four curve segments, forming partial Turing bifurcation curve $l^T$ (see Figure 12a). From Theorem 9, if we choose $d_1$ and $d_2$ in the region above the curve, the constant steady state $E^{*}$ is stable. If we choose $d_1=1.496$ and $d_2=0.000688$ in the region under $l^T$, the steady state $E^{*}$ of the diffusive system is Turing unstable, and there is a spatially inhomogeneous steady state (see Figure 13).

Figure 12. (a) Partial Turing bifurcation curve $l^T$ of system (4) with no delays with the parameters $a=0.23,r=1.1,m=0.31,l=2,c=0.25<\frac{1}{r(1-a)},p=1.4<p_H$. (b) When $a=0.23,r=1.1,$$m=0.31,l=2,c=0.25,d_1=1.496$, the Turing bifurcation curve $d_2=d_2(5,d_1)$ intersects line $p=p_H$ at $TH (1.5432,0.0009)$, which is a Turing–Hopf bifurcation point.

Figure 13. (a) The prey pattern and (b) the predator pattern for system (4) without delays with the parameters $a=0.23,r=1.1,m=0.31,l=2,c=0.25,p=1.4,d_1=1.496,d_2=0.000688$. The patterns indicate instability induced by diffusion for the prey and predator. Initial conditions are $u(x,0)=0.6+0.1cos2x$, $v(x,0)=0.08-0.02cos2x$.

If we fix $d_1=1.496$, from (16), we can draw Turing bifurcation curve on $p-d_2$ plane (see Figure 12b), and $d_2=d_2^T(5,d_1)$ intersects line $p=p_H$ at $TH (1.5432,0.0009)$. Choosing $p=1.5432,d_2=0.00081$ near $TH$, two stable spatially inhomogeneous periodic solutions coexist when we choose two different initial values (see Figure 14).

Figure 14. Two spatially inhomogeneous periodic solutions coexist when we choose $a=0.23,m=0.31,$$r=1.1$, $l=2$, $c=0.25$, $p=p_H=1.5432$, $d_1=1.496$, and $d_2=0.00081$ in system (4) without delays. Initial conditions are $u(x,0)=0.6+0.2cos2x$, $v(x,0)=0.08-0.01cos2x$ for (a,b), and $u(x,0)=0.6+0.2cos3x$, $v(x,0)=0.08-0.01cos3x$ for (c,d).

To illustrate the dynamics of (4) without delays in the case of strong cooperation, we choose $c=8$ such that $c>\frac{1}{r(1-a)}$. Let $p=0.8<p_H=0.8306$. From (16) and (17), we can draw partial Turing bifurcation curve $l^T$ on $d_1-d_2$ plane (see Figure 15a). The constant steady state $E^{*}(0.7392,0.0864)$ is stable when we choose $d_1$ and $d_2$ in the region above $l^T$.

Figure 15. (a) Partial Turing bifurcation curve $l^T$ of system (4) without delays with the parameters $a=0.23,r=1.1,m=0.31,l=2,c=8,p=0.8<p_H$. (b) When $a=0.23,r=1.1,m=0.31,l=2,$$c=8,d_1=0.13$, Turing bifurcation curve $d_2=d_2(5,d_1)$ intersects line $p=p_H$ at $TH (0.8306,0.009)$, which is a Turing–Hopf bifurcation point.

If we choose $d_1=0.13$ and $d_2=0.006$ in the region below $l^T$, from Theorem 9, the steady state $E^{*}$ of the diffusive system is Turing unstable. If we fix $d_1=0.13$, from (16), we can draw Turing bifurcation curve on $p-d_2$ plane (see Figure 15b), and $d_2=d_2^T(5,d_1)$ intersects line $p=p_H$ at $TH (0.8306,0.009)$. Choosing $p=0.8306,d_2=0.0065$ near $TH$, we can illustrate two stable spatially inhomogeneous periodic solutions coexisting when we choose two different initial values (see Figure 16).

Figure 16. Two stable spatially inhomogeneous periodic solutions of system (4) without delays coexist with $a=0.23,m=0.31,r=1.1$, $l=2$, $c=8$, $p=p_H=0.8306$, $d_1=0.13$, and $d_2=0.0065$. Initial values are $u(x,0)=0.6+0.1cos4x$, $v(x,0)=0.08-0.02cos4x$ for (a,b), and $u(x,0)=0.6+0.1cos3x$, $v(x,0)=0.08-0.02cos3x$ for (c,d).

6.3. Numerical Simulations for Diffusive System with Two Delays

In this section, we carry out some numerical simulations for diffusive system (4). Fix

$$a=0.23,r=1.1,m=0.31,l=2,$$

which is the same as in (31). Fix $c=0.25$ such that $c<\frac{1}{r(1-a)}$, and choose $p=1.2<p_H$, $d_1=0.3$, and $d_2=0.4$, such that $d_2>d_2^T(1,d_1)$ for all $n\in \mathbb{N}$. From Theorem 9, one can get the unique constant steady state $E^{*}(0.80945,0.11797)$ is locally asymptotically stable.

Now, we illustrate the effect of two delays on the dynamics (4). Following the steps in Appendix C and Appendix D, we can draw all the stability switching curves on $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ plane, and decide the crossing direction, which are shown in Figure A3, Figure A4, Figure A5, Figure A6, Figure A7 in Appendix E. Combining all the stability switching curves together, and zooming in the part of $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)\in [0,5]\times [0,20]$, we get the Hopf bifurcation curves shown in Figure 17a. Consider the bottom left region bounded by left-most curve of ${\mathcal{T}}_0^1$ and the lowest curve of ${\mathcal{T}}_0^2$ (see Figure 17a), which are both part of ${\mathcal{T}}_0$. The crossing directions of the two switching curves (the black line and blue line) are all pointing outside of the region. From Theorem 9, under the assumption $p<p_H$ and $d_2>d_2^T(n,d_1)$, $E^{*}$ is locally asymptotically stable. From Theorem 10, when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross from the left to the right of the stability switching curve ${\mathrm{\tau }}_0^{2\left(1\right)}$ (or cross from the bottom to the top of ${\mathrm{\tau }}_0^{1\left(1\right)}$), the characteristic equation has two more characteristic roots with positive real parts from Lemma A1, $E^{*}$ becomes unstable, and periodic solution occurs due to Hopf bifurcation. Similarly, when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross from the left to the right of stability switching curve ${\mathrm{\tau }}_1^{1\left(1\right)}$ (or from the left to the right of stability switching curve ${\mathrm{\tau }}_2^{1\left(1\right)}$ ), the characteristic equation has two more characteristic roots. Then, $E^{*}$ is still unstable, and system (4) undergoes Hopf bifurcation and periodic solution arises. However, the periodic solution is unstable, as there is unstable manifold. It means that stability switching curves ${\mathrm{\tau }}_0^{2\left(1\right)}$ and ${\mathrm{\tau }}_0^{1\left(1\right)}$ are the boundary of stable region for $E^{*}$. In the stable region in Figure 17a, the predator and the prey coexist, and when $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ cross the stability switching curves ${\mathrm{\tau }}_0^{2\left(1\right)}$ (or ${\mathrm{\tau }}_0^{1\left(1\right)}$), the species coexist in the form of sustained fluctuation. From the previous discussion, if we choose a point (2.31, 0) on ${\mathrm{\tau }}_1-$axis and move up, system (4) undergoes stability switches at $E^{*}$. There are two intersections with the boundary of the stable region: $(2.31,3.25)$, $(2.31,10.41)$ (see Figure 17a). The positive equilibrium $E^{*}$ is unstable when ${\mathrm{\tau }}_2\in [0,2.31)$ and there are periodic solutions; $E^{*}$ becomes locally asymptotically stable and periodic solutions disappear when ${\mathrm{\tau }}_2\in (2.31,10.41)$, and $E^{*}$ loses its stability when ${\mathrm{\tau }}_2>10.41$ and periodic solution appears again. Multiple times stability switching means that the stability of $E^{*}$ can switch with the varying of parameters $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$, and periodic solutions may appear or disappear during the stability switching of $E^{*}$.

Figure 17. (a) The left-most curve and the lowest curve of ${\mathcal{T}}_0$ intersect at $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)=(2.2140,15.0019)$, which is a double Hopf bifurcation point on the ${\mathrm{\tau }}_1-{\mathrm{\tau }}_2$ plane. Crossing directions are marked by arrows. (b) The complete bifurcation sets near double Hopf bifurcation point HH.

Moreover, the two stability switching curves intersect at the point $(2.21407,15.0019)$, which is the double Hopf bifurcation point, denoted by HH. For HH, ${\omega }_1=0.1078$, ${\omega }_2=0.4920$. Using the normal form derivation process given in Supplementary Materials, we have $B_{11}=-0.0512$, $B_{21}=0.0478$, $B_{13}=1.7736$, $B_{23}=-0.0706$, $B_{2100}=-0.7294$, $B_{1011}=-1.5826$, $B_{0021}=-0.2410$, $B_{1110}=-10.8681$. Furthermore, we have the normal form (29) with ${ϵ}_1=-1,{ϵ}_2=-1,b=6.5658,c=14.8998,d=1$, and $d-bc=-96.8306$. According to Chapter 7.5 in [62], case Ib arises, and we have the bifurcation set near HH showing in Figure 17b. In region $D_1$, the positive equilibrium is asymptotically stable. In region $D_2$ or $D_6$, there is a stable periodic solution. When the parameters cross into the region D4, there are two stable homogeneous periodic solutions coexisting in D4.

7. Concluding Remarks

We investigate the dynamics of a diffusive predator–prey model with two delays. First, we discuss the existence of constant coexisting steady states of system (4). It turns out that the strength of cooperation $c=\frac{1}{r(1-a)}$ is a threshold: if $c<\frac{1}{r(1-a)}$, i.e., the cooperation is weak, there is at most one coexisting steady state, and if $c>\frac{1}{r(1-a)}$, i.e., the cooperation is strong, there are at most two coexisting steady states.

To figure out the dynamics of system (4), we first carry out the study on the kinetic predator–prey model (7) with Allee effect in prey and cooperative hunting in predators. Compared with the work of Jang et al. [17], which mainly gives the local bifurcation analysis, we mainly focus on the global dynamics. We start with the kinetic system in the case of weak cooperation. Taking p, the conversion rate, as the bifurcation parameter, we prove that there is a loop of heteroclinic orbits between $E_1$ and $E_a$ formed by the intersection of the stable manifold of $E_a$ and the unstable manifold of $E_1$ when $p=p^{\#}$ ($p^{\#}>1$). It shows that $p^{\#}$ is an important threshold, which distinguishes the globally stability for $E_0$ and the existence of a separatrix curve. To be specific, when $p>p^{\#}$, $E_0$ is globally stable, which means the extinction of both species, and we call the phenomenon overexploitation. When $p<p^{\#}$, there is a separatrix curve ${\mathrm{\Gamma }}_p^s$, which is determined by the stable manifold of $E_a$. High initial predator density will lead to extinction for both species, and conversely low initial predator density will approach either a steady state or a periodic oscillation. When $p<1$, $E_1(1,0)$ is locally stable, which means that the prey achieves its carrying capacity and the predator is extinct. When $1<p<p_H$, the stable state becomes $E^{*}$, which implies the coexistence of both species. When p crosses $p_H$, the stable state switches into a limit cycle. The period of the limit cycle increases with p, and tends to ∞ as $p\to p^{\#}$. The existence of a limit cycle with a large amplitude increases the risk of the extinction for the predator.

In the case of strong cooperation, three different cases are considered. We prove the existence of a loop of heteroclinic orbits between $E_1$ and $E_a$ when $p=p^{\#}$ in the first and second cases and nonexistence of such a loop of heteroclinic orbits in the third case. We also exhibit the existence of loop of heteroclinic orbits among $E_R^{*}$, $E_a$, and $E_1$ formed by the unstable manifold of $E_R^{*}$ and the unstable manifold $E_a$ when $p={\overline{p}}^{\#}$, and a homoclinic cycle when $p=p_{hom}$ by numerical simulations in case 3. When $p>p^{\#}$ in case 1 and case 2 ($p>{\overline{p}}^{\#}$ in case 3), both species are distinct. When $1<p<p^{\#}$ in case 1 and case 2, there is a separatrix curve ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, and there are two stable states coexisting, which are extinction for both species and coexistence or oscillation. When $p<p^{\#}$ (or $p<{\overline{p}}^{\#}$) and $p_{SN}<p<1$, there are two separatrix curves, the stable manifold of $E_a$ and the stable manifold of $E_R^{*}$, separating the first quadrant into three parts. If the initial population is above the stable manifold of $E_a$, both species are extinct; if the initial population is inside the stable manifold of $E_R^{*}$, the stable state can be $E^{*}$ or limit cycle; if the initial population is below the stable manifold of $E_a$ and outside the stable manifold of the stable manifold of $E_R^{*}$, the prey achieves its carrying capacity and the predator is extinct. When $p<p_{SN}$, there is one separatrix curve ${\mathrm{\Gamma }}_p^s\left(E_a\right)$, and there are two stable states coexisting. The species with initial population above the separatrix finally become extinct, however only the prey survive when initial population is below it.

There is a unique equilibrium for weak cooperation when $1<p<\frac{1}{a}$. It is shown that there are at most two stable states coexist for different initial values separated by the stable manifold of $E_a$. There are two interior equilibria $E^{*}$ and $E_R^{*}$ in the case of strong cooperation when $p_{SN}<p<1$. The equilibrium $E_R^{*}$ brings one more separatrix curve than the case of weak cooperation, thus there may be three stable states coexisting.

Second, we consider the diffusive system without delays, and focus on the effect of diffusion on the dynamics. Taking diffusion coefficients $d_1$ and $d_2$ as bifurcation parameters, we give the conditions of existence of Turing instability and Turing–Hopf bifurcation for the system with both weak and strong cooperation. We illustrate complex dynamics of the diffusive system, including the existence of spatially inhomogeneous steady state, coexistence of two spatially inhomogeneous periodic solutions.

Finally, we discuss the joint effect of two delays on the dynamical behavior of the diffusive system. Applying the method of stability switching curves, we find all the stability switching curves at which the characteristic roots are purely imaginary. Combining the geometrical and analytic method, we can decide the crossing direction of the characteristic roots as long as we confirm the positive direction for stability switching curves. Then, we can get the condition of existence of Hopf bifurcation. By searching the intersection of stability switching curves near the stable region, we get the double Hopf bifurcation point. Through the calculation of normal form of the system, we get the corresponding unfolding system and the bifurcation set, from which we can figure out the complete dynamics near the bifurcation point on the $({\mathrm{\tau }}_1,{\mathrm{\tau }}_2)$ parametric plane. We theoretically prove and illustrate that near the bifurcation point, there are the phenomena of the stability of positive equilibrium: stable periodic solutions and the coexistence of two periodic solutions. From double Hopf bifurcation analysis, we can gain an insight into the local dynamics near the double Hopf bifurcation point. However, numerical simulations tells us that the occurrence of complex phenomena near bifurcation point is sensitive to the choice of initial values. When we choose the initial values with high ratio of predator to prey, both species are extinct. Thus, the complex phenomena near the double Hopf bifurcation probably accompany with an extinction steady state, which is consistent with the results in the kinetic system. Therefore, the analysis of global dynamics of kinetic system can help us comprehend these bistable dynamics for the diffusive system.

Appendix A.1. The Properties of Stable and Unstable Manifold and the Existence or Nonexistence of Periodic Orbits

Proposition A1.

Suppose that $c<\frac{1}{r(1-a)}$, and $1<p<\frac{1}{a}$.

 (i)

The orbit ${\mathrm{\Gamma }}_p^s$ meets the v-nullcline $v=g\left(u\right)$ at a point $(u_p^s,S\left(p\right))$, where $S\left(p\right)\ge v^{*}:=f\left(u^{*}\right)$, and $S\left(p\right)$ is a monotone decreasing function for $p\in (1,\frac{1}{a})$.

 (ii)

The orbit ${\mathrm{\Gamma }}_p^u$ meets the v-nullcline $v=g\left(u\right)$ at a point $(u_p^u,U\left(p\right))$, where $U\left(p\right)\ge v^{*}:=f\left(u^{*}\right)$, and $U\left(p\right)$ is a monotone increasing function for $p\in (1,\frac{1}{a})$.

Proof. 

Inspired by the authors of [29], we prove (i), and the proof for (ii) is similar.

When $p<\frac{1}{a}$, $E_a$ is a saddle. The eigenvector corresponding to ${\lambda }_1=ra(1-a)>0$ is $(1,0)$, which means that the unstable manifold of $E_a$ is on the u-axis. For ${\lambda }_2=m(pa-1)<0$, the corresponding eigenvector is ${(1,r(1-a)-\frac{m(pa-1)}{a})}^T$. Thus, the tangent vector of the stable manifold of $E_a(a,0)$ (denoted by ${\mathrm{\Gamma }}_p^s$) at $E_a(a,0)$ is $k_1=r(1-a)-\frac{m(pa-1)}{a}$. Comparing with the tangent vector $k_2=r(1-a)$ of $v=f\left(u\right)$ at $E_a(a,0)$, we have $k_1>k_2$, thus ${\mathrm{\Gamma }}_p^s$ is above the nullcline $v=f\left(u\right)$ near $E_a$. Moreover, from the vector field in (7) on $v=f\left(u\right)$, ${\mathrm{\Gamma }}_p^s$ is always above $v=f\left(u\right)$ before ${\mathrm{\Gamma }}_p^s$ meets the v-nullcline.

In order to show that ${\mathrm{\Gamma }}_p^s$ meets the v-nullcline, we still need to prove that it remains bounded for $u>a$. In fact, for $u>a$, ${\mathrm{\Gamma }}_p^s$ is the graph of a function $v\left(u\right)$, satisfying

$$\frac{\mathrm{d}v\left(u\right)}{\mathrm{d}u}=\frac{mv[1-pu(1+cv\left)\right]}{u\left[\right(1+cv)v-r(1-u\left)\right(u-a\left)\right]}.$$

If there is a $u_b<u^{*}$ such that $v\left(u\right)\to \infty$ as $u\to u_b^{-}$, then $(1+cv)v-r(1-u)(u-a)$ is bounded below for $a+\epsilon \le u\le u_b$ and any $\epsilon >0$. Thus, for $a+\epsilon \le u\le u_b$,

$$\frac{\mathrm{d}v}{\mathrm{d}u}\le Av$$

for some positive constant A. It means that $v\left(u\right)$ is bounded as $u\to u_b^{-}$, which is a contradiction. Thus it cannot blow up before it extends to the v-nullcline $v=g\left(u\right)$. Therefore, $S\left(p\right)$ exists for all $p\in (1,\frac{1}{a})$ and $S\left(p\right)\ge v^{*}$.

Notice that $S\left(p\right)=v^{*}$ only when ${\mathrm{\Gamma }}_p^s\to E^{*}$ as $t\to -\infty$, which means $E^{*}$ must be an unstable node. When $1<p<p_H$, $E^{*}$ is locally stable, thus $S\left(p\right)>v^{*}$ if $p<p_H$. When $p_H<p<\frac{1}{a}$ and near $p_H$, $E^{*}$ is an unstable spiral. Thus, the set $K=\{p\in (1,\frac{1}{a}):S\left(p\right)>v^{*}\}$ is a nonempty open set containing $(1,p_H+\epsilon )$.

Now we show that $S\left(p\right)$ is monotonically decreasing on any component of K. Let $p_1<p_2$ be two points in some interval in K. Denote the v-nullcline for $p_1$ and $p_2$ as $v=g_1\left(u\right)$ and $v=g_2\left(u\right)$. From the expression of $v=g\left(u\right)$, we know that the curve of $v=g_1\left(u\right)$ is on the right of $v=g_2\left(u\right)$. The stable manifold of $E_a(a,0)$ for $p_1$ and $p_2$ are graphs of function $v_1\left(u\right)$ and $v_2\left(u\right)$ defined for $u>a$, respectively. The corresponding eigenvectors at $E_a$ are $x_1={(1,r(1-a)-\frac{m(p_{1}a-1)}{a})}^T$, $x_2={(1,r(1-a)-\frac{m(p_{2}a-1)}{a})}^T$, respectively. Therefore, $v_1\left(u\right)>v_2\left(u\right)$ for u sufficiently near a. Suppose that $v_1\left(u\right)=v_2\left(u\right)$ for some u with $a<u\le \frac{1}{p_2}$. Let $\overline{u}$ be the smallest such value. Then, we must have $v_2^{\prime }\left(\overline{u}\right)\ge v_1^{\prime }\left(\overline{u}\right)\ge 0$. However, $\frac{m{v}_1\left(\overline{u}\right)[1-p_1\overline{u}(1+c{v}_1\left(\overline{u}\right))]}{\overline{u}[(1+c{v}_1\left(\overline{u}\right))v_1\left(\overline{u}\right)-r(1-\overline{u})(\overline{u}-a)]}\le \frac{m{v}_2\left(\overline{u}\right)[1-p_2\overline{u}(1+c{v}_2\left(\overline{u}\right))]}{\overline{u}[(1+c{v}_2\left(\overline{u}\right))v_2\left(\overline{u}\right)-r(1-\overline{u})(\overline{u}-a)]}$ implies that $p_1\ge p_2$, which is a contradiction. Thus, the intersection of $v_2\left(u\right)$ and $g_2\left(u\right)$, $S\left(p_2\right)$, is below the intersection of $v_1\left(u\right)$ and $g_2\left(u\right)$. For $v_1\left(u\right)$ between $g_2\left(u\right)$ and $g_1\left(u\right)$, from the vector field on the left of $g_1\left(u\right)$, the intersection of $v_1\left(u\right)$ and $g_1\left(u\right)$, $S\left(p_1\right)$, is higher than that of $v_1\left(u\right)$ and $g_2\left(u\right)$. Thus, $S\left(p_2\right)<S\left(p_1\right)$. Notice that the argument above shows that if $p_2\in K$ and $S\left(p_2\right)>v^{*}$, then any $p\in (1,p_2)$ also belongs to K and $S\left(p\right)>S\left(p_2\right)$.

We can claim now $K=(1,p_a)$ for some $p_a\in (p_H,\frac{1}{a})$. For $p\in (1,p_a)$, $S\left(p\right)>v^{*}$, and for $p\in [p_a,\frac{1}{a})$, $S\left(p\right)=v^{*}$. It remains to prove $S\left(p\right)$ is decreasing when $p\in [p_a,\frac{1}{a})$. From the vector field in (7), ${\mathrm{\Gamma }}_p^s$ moves towards the upper right, backward, before it meets the v-nullcline. Then, for $p=p_{\mathrm{top}}$, we have $S\left(p\right)>v^{*}$, and thus $p_{\mathrm{top}}<p_a$. It is obvious that $v^{*}$ is decreasing with respect to p for $p\in (p_{\mathrm{top}},\frac{1}{a})$, and thus so does for $p\in [p_a,\frac{1}{a})$. Therefore, $S\left(p\right)$ is decreasing for both $(1,p_a)$ and $[p_a,\frac{1}{a})$. □

Obviously, $p^{\#}$ is a threshold value of the property of ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$. When $p<p^{\#}$, ${\mathrm{\Gamma }}_p^s$ is above ${\mathrm{\Gamma }}_p^u$, and when $p>p^{\#}$, ${\mathrm{\Gamma }}_p^s$ is below ${\mathrm{\Gamma }}_p^u$. Let $\mathrm{\Omega }$ denotes the bounded open subset of the positive quadrant, bounded by ${\mathrm{\Gamma }}_p^s$, v-nullcline between ${\mathrm{\Gamma }}_p^s$ and ${\mathrm{\Gamma }}_p^u$, ${\mathrm{\Gamma }}_p^u$, and the segment from $E_1$ to $E_a$ on the $u-$axis (see Figure A1).

Figure A1. The dashed curve is the u-nullcline, and the dotted curve is the v-nullcline. Different positions of the stable manifold ${\mathrm{\Gamma }}_p^s$ of $E_a$ and the unstable manifold ${\mathrm{\Gamma }}_p^u$ of $E_1$ for (a) $1<p<p^{\#}$ and (b) $p^{\#}<p<\frac{1}{a}$.

Proposition A2.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

If $1<p<p^{\#}$, then $S\left(p\right)>U\left(p\right)$. Moreover, all orbits in the positive quadrant above ${\mathrm{\Gamma }}_p^s$ converge to $E_0$, and all orbits below ${\mathrm{\Gamma }}_p^s$ have their $\omega -$limit sets in Ω, which is a positive invariant set.

 (ii)

If $p^{\#}<p<\frac{1}{a}$, then $S\left(p\right)<U\left(p\right)$, and ${\mathrm{\Gamma }}_p^u$ tends to $E_0$. Moreover, all orbits in the positive quadrant above ${\mathrm{\Gamma }}_p^u$ converge to $E_0$, and all orbits below ${\mathrm{\Gamma }}_p^u$ have their $\alpha -$limit sets in Ω, which is a negative invariant set.

Proof. 

We prove the case for $1<p<p^{\#}$, and the other case can be proved similarly. Propositions A1 and 2 directly lead to $S\left(p\right)>U\left(p\right)$. From Proposition A1, ${\mathrm{\Gamma }}_p^s$ enters $E_a$ from the region above the u-nullcline $v=f\left(u\right)$, and meets the v-nullcline at $(u_p^s,S\left(p\right))$. On the right of $(u_p^s,S\left(p\right))$, ${\mathrm{\Gamma }}_p^s$ is still above $v=f\left(u\right)$, since ${\mathrm{\Gamma }}_p^u$ is above the u-nullcline and $S\left(p\right)>U\left(p\right)$. Moreover, ${\mathrm{\Gamma }}_p^s$ can not tend to $E_1$. In fact, from the direction of the vector field in system (7), ${\mathrm{\Gamma }}_p^s$ goes to the lower right as $t\to -\infty$. Thus, ${\mathrm{\Gamma }}_p^s$ divides the first quadrant into two regions. Noticing that the first quadrant is invariant, we can get the results from Poncaré–Bendixson theorem. □

Moreover, Poincaré–Bendixson theorem yields the following conclusion.

Proposition A3.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

If $1<p<p^{\#}$, then either $E^{*}$ is stable or Ω contains a periodic orbits which is stable from the outside (both may be true).

 (ii)

If $p^{\#}<p<\frac{1}{a}$, then either $E^{*}$ is unstable or Ω contains a periodic orbits which is unstable from the outside (both may be true).

From Proposition A3, in order to figure out the dynamics in $\mathrm{\Omega }$ in detail, we have to consider the existence and nonexistence of periodic orbits. In Section 3.1.1, we have discussed the existence of periodic orbits bifurcating from Hopf bifurcation. Now, we consider the nonexistence of periodic orbits.

Proposition A4.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

There is an ${\epsilon }_1>0$ such that if $\frac{1}{a}-{\epsilon }_1<p<\frac{1}{a}$, there is no periodic orbits, and ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, which is a heteroclinic orbit.

 (ii)

There is an ${\epsilon }_2>0$ such that if $1<p<1+{\epsilon }_2$, there is no periodic orbits, and ${\mathrm{\Gamma }}_p^u$ connects $E_1$ to $E^{*}$.

Proof. 

We prove the case for $p<\frac{1}{a}$ and near $\frac{1}{a}$, and the latter case can be proved similarly. If $S\left(p\right)=v^{*}$, ${\mathrm{\Gamma }}_p^s$ connects $E^{*}$ to $E_a$, and there is no periodic orbits. When $S\left(p\right)>v^{*}$, denote the intersection of ${\mathrm{\Gamma }}_p^s$ and v-nullcline as $P(u_p^s,S\left(p\right))$. It is easy to confirm that there is a $p_1$ such that $S\left(p\right)=f\left(u_{p_1}\right)$, where $f\left(u_{p_1}\right)$ is the component of the intersection of u-nullcline and v-nullcline corresponding to $p_1$ (denoted by $P_1(u_{p_1},f\left(u_{p_1}\right)$). Consider the region with vertices $E_a$, P, $P_1$, $Q_1(\frac{1}{p_1},0)$, which is a negative invariant region. $P_1$ is well defined when p is close to $\frac{1}{a}$ as $S\left(p\right)<f\left(p_{top}\right)$. As $E^{*}$ is the unique equilibrium in this region, thus if there are periodic orbits, they must encircle $E^{*}$ and lie wholly in this region. However, the divergence of the vector field of (7) is positive for $p\to {\frac{1}{a}}^{-}$, as it is $ra(1-a)+m(pa-1)$ at $E_a$, which is positive. From Bendixson’s criterion, there is no periodic orbits in the region. Therefore, according to Poincaré Bendixson theorem, $E^{*}$ is the $\alpha -$limit set of ${\mathrm{\Gamma }}_p^s$. □

Theorem 3 provides a global description of the dynamical behaviour of system (7) for $p\in {\mathbb{R}}^{+}\backslash (1+{\epsilon }_2,\frac{1}{a}-{\epsilon }_1)\stackrel{∆}{=}{\mathbb{R}}^{+}\backslash I$. When $p\in I$, we can go further with the method in [30].

Proposition A5.

Suppose that $c<\frac{1}{r(1-a)}$.

 (i)

If $p_H<p^{\#}$, then

 (a)

for $p_H<p<p^{\#}$, there is at least one periodic solution in Ω which is stable from the outside and one (perhaps the same) stable from inside;

 (b)

if $a\left(p_H\right)>0$, then there is an $\epsilon >0$ such that if $p_H-\epsilon <p<p_H$, there are at least two distinct periodic solutions, the inner of which is unstable while the outer is stable from the outside.

 (ii)

If $p^{\#}<p_H$, then

 (a)

for $p^{\#}<p<p_H$, there is at least one periodic solution in Ω which is unstable from the outside and one (not necessarily distinct) unstable from inside;

 (b)

if $a\left(p_H\right)<0$, then there is an $\epsilon >0$ such that if $p_H<p<p_H+\epsilon$, there are at least two distinct periodic solutions, the inner is stable while the outer is unstable from the outside.

Proof. 

We prove (i), and (ii) can be proved analogously. If $p_H<p<p^{\#}$, $E^{*}$ is a repellor. As $\mathrm{\Omega }$ is a positive invariant region, ($i_1$) follows from the Poincaré–Bendixson theorem. If $a\left(p_H\right)>0$, an unstable periodic solution bifurcates from $E^{*}$ as p decreases past $p_H$. Again $\mathrm{\Omega }$ is a positive invariant region, from Poincaré–Bendixon theorem, there is a second periodic orbit exterior to the unstable bifurcating periodic orbit. □

Appendix A.2. The Proof of Theorem 3

Proof.(i)

If $p\ge \frac{1}{a}$, there is no interior equilibrium, then there is no periodic orbit in the first quadrant. Thus, every orbit converges to a boundary equilibrium. We have known that $E_0$ is a stable node, $E_1$ is a saddle, and $E_a$ is a unstable node if $p>\frac{1}{a}$ and a non-hyperbolic repellor if $p=\frac{1}{a}$. Noting that the first quadrant is positive invariant, then $E_0$ is globally asymptotically stable.

(ii) and (iii) follows from Propositions A2 and A4 and the fact that $E^{*}$ is a sink when $p>1$ and close to 1 and a source when $p<\frac{1}{a}$ and close to $\frac{1}{a}$.

(iv) If p decreases to $p=1$, $E^{*}$ collides with $E_1$. A transcritical bifurcation involving $E^{*}$ and $E_1$ occurs. $E_1$ changes its stability from a saddle point to a stable node. If $p<1$, there is no interior equilibrium, and there is no periodic orbit in the first quadrant. $E_0$ and $E_1$ are both stable node. $E_a$ is a saddle, and its stable manifold ${\mathrm{\Gamma }}_p^s$ divides the first quadrant into two regions. The region above ${\mathrm{\Gamma }}_p^s$ is the attractive basin of $E_0$, and the region below ${\mathrm{\Gamma }}_p^s$ is the attractive basin of $E_1$. □

Appendix A.3. Proof of Theorem 4

Proof. 

If $a\left(p_H\right)<0$, from Theorem 2, stable periodic orbits bifurcating from Hopf bifurcation appear when $p>p_H$. According to Proposition A5 (ii), if $p_{\#}<p_H$, there will be at least two distinct periodic orbits, which contradicts our assumption about the uniqueness of periodic orbit. Thus, we have $p_H\le p_{\#}$. As $E^{*}$ is asymptotically stable for any $1<p<p_H$, if there is a periodic orbit, it must be unstable from inside, which contradicts our assumption about the orbital stability of periodic orbit. Therefore, there is no periodic orbit when $1<p<p_H$, and the conclusion (i) follows from Proposition A2 (i).

For the case of $p^{\#}<p<\frac{1}{a}$, any orbit must encircle $E^{*}$ and lie wholly in $\mathrm{\Omega }$, which is negatively invariant by Proposition A2 (ii). Thus, if there is a periodic orbit, it must be unstable from outside, which contradicts our assumption about the orbital stability of periodic orbit. Therefore, the conclusion (iv) follows from Proposition A2 (ii). Moreover, together with the fact that Hopf bifurcating periodic orbit appears when $p>p_H$, the nonexistence of periodic orbits for $p^{\#}<p<\frac{1}{a}$ implies that $p_H$ is strictly less that $p_{\#}$.

If $p_H<p<p^{\#}$, $E^{*}$ is a repellor, thus it follows from Proposition A3 (i) that there is a periodic orbit for all $p_H<p<p^{\#}$. Noticing that we have proved there is no cycle for $p>p^{\#}$, according to the work of [68,69,70], the period of the unique cycle must tend to infinity as $p\to p^{\#}$. Proposition A2 (i) completes the proof of (ii).

When $p=p^{\#}$, then $p>p_H$, and $E^{*}$ is a repellor, from Poincaré–Bendixson theorem, the loop of heteroclicic orbits is the $\omega$-limit set of orbits through any point in its interior. □