The Evolutionary Dynamics of a Sex-Structured Population with Non-Overlapping Generations

Abstract

This paper proposes and studies a discrete-time model for a sex-structured population with non-overlapping generations under density-dependent regulation of survival. The population is assumed to have genetic variety among individuals in terms of reproductive potential, controlled by a single autosomal diallelic locus. We consider a panmictic population with Mendelian inheritance rules. We examine the stability model and show that increasing the average value of reproductive potential destabilizes the population dynamics. The scenario of stability loss in fixed points via period doubling or Neimark–Sacker bifurcations depends on the intensity of the self-regulation. The growth rate at which the population survives and develops is shown to depend on the fitness of the genotypes and the secondary sex ratio. As a result, the asymptotic genetic composition of the population is determined by the values of the reproductive potentials of the heterozygote and homozygotes, the initial conditions, and the parameter describing the ratio of newborn females to males. With disruptive selection, the influence of external factors changing the current genetic composition of a population can alter the direction of evolution and lead to the extinction of a successful developing population or a gradual population recovery due to evolutionary rescue after a noticeable decline in its abundance.

1. Introduction

The fundamental principles of general biology that pertain to population evolution and ecosystem dynamics were established under the assumption that ecological and evolutionary processes are interconnected and that their interactions can be significant. In particular, the author of the theory of evolution, Charles Darwin (Darwin, 1859), the creators of the synthetic “genetic” theory of evolution, R. Fisher (1930), S. Wright (1930), J. Haldane (1932), and S.S. Chetverikov (1926), as well as the founders of theoretical ecology, P. Verhulst (1847) and V. Volterra (1928) (as well as G.F. Gause (1935)) [1,2,3,4,5,6,7,8], considered evolutionary and ecological processes as integral characteristics of a whole. Additionally, the study of ecological and evolutionary dynamics assumes that organisms’ life cycle features can evolve on a similar timescale as their population dynamics [9,10,11]. On the one hand, changes in genetic structure occurring during the evolution of a population (primarily under the influence of natural selection) affect the birth rate and, accordingly, its dynamics. On the other hand, as a rule, fluctuations in abundance are caused by changes in the population structure caused by the combined effects of environmental factors and self-regulatory mechanisms [12,13,14]. Therefore, when modeling population dynamics, it is advisable to use an ecological–genetic approach that allows us to study the interconnected changes in the abundance and genetic structure of a population [15,16,17,18,19,20,21,22,23,24,25,26,27].

Within the approach combining classic population genetic theory and ecological modeling, the dynamics of genetic frequencies are studied depending on various inheritance patterns and using the relative fitness values of genotypes, which allows for considering the population abundance to be constant without analyzing its dynamics [2,28]. This approach remains relevant today (e.g., [29,30,31]). A modification of unstructured ecological models by including genotype fitness as a function of intra- and interspecies density [16] further develops this approach. In parallel, population genetics extends by considering both age [32,33,34] and sex [35] structures. The concepts of density- and frequency-dependent components in the action of natural selection have been developed [32,33,34], as have density-independent forms of natural selection, whose action under ecological limitation can cause a change in the nature of the population abundance dynamics [36,37,38]. Currently, research continues to investigate the relationship between population genetics, stage (age) [23,24,25,26,39], and sex [22] structures. The work [22] derives conditions for protected polymorphism in a population characterized by age- or stage-dependent demography with two sexes.

Additionally, within the synthetic theory, there are studies using simplified models of evolution that do not consider genetic dynamics in detail, i.e., they do not imply a detailed description of the inheritance mechanism of the traits under consideration. In such studies, as a rule, the evolution of polygenic and continuously distributed traits (or quantitative traits [40]) is modeled (e.g., [41,42,43,44,45,46]), as is the evolution of clonal systems in which individuals have separate discrete trait values and ideal inheritance of phenotypes, disturbed by rare mutations [41,44,46,47,48]. In general, despite the large number of papers and diversity of studied objects, unresolved issues remain related to the dynamics and evolutionary features of structured populations and the study of their regulatory mechanisms.

On the other hand, combining classic population genetic theory and ecological modeling makes it possible to see the effects that are missed when using each of these approaches in isolation, as well as to study the interconnected changes in the abundance and genetic structure of a population under ecological limitation during the influence of natural selection [20,21,22,23,24,25,26,27,49,50,51,52]. Examples include the effects of bottlenecks, evolutionary rescue, or degeneration, in which genotype ratios determine the growth or decline of population abundance. Note that sometimes there is a catastrophic decrease in population size up to several reproductive pairs (individuals), which are subject to the threat of extinction. A small population that is in danger of extinction still has two possible development options: to become extinct or to adapt with a subsequent recovery of the population size after passing through a “bottleneck” [53]. Note that massive non-selective mortality allows only a few individuals to pass on their genes to subsequent generations. Consequently, a temporary reduction in population size (bottlenecks in population size) can affect the genetic variability, mutation load, and inbreeding levels in populations [53,54,55,56]. When endangered species manage to adapt to changes in the environment within sufficiently short time frames to prevent their extinction, this is referred to as evolutionary rescue [56,57,58]. Examples of natural populations that have temporarily passed through a bottleneck include the following: Mexican and red wolfs [59], island gray Californian foxes [60], Mednyi arctic fox [55], northern elephant seal [61], European bison [62], and cheetah [63]. The natural interconnection of ecological and genetic processes and the discovered effects that are observed in biological populations confirm the promise and importance of developing eco-evolutionary approaches. Maintaining genetic polymorphism is a crucial issue in population genetics [22,23,24,28,52,64,65,66,67,68,69], since a reduction in genetic variety combined with genetic drift and inbreeding can lead to unpredictable and even catastrophic consequences for the population. Thus, among the main results of population genetics are the criteria that determine whether natural selection leads to the fixation of one genotype or the coexistence of several genotypes with polymorphism [28,30,47,65,66,67,68,70]. For example, the lack of genetic variety in cheetahs led to abnormalities in sperm formation, reduced fertility, high infant mortality, and increased susceptibility to diseases [71].

The present work is devoted to the development of an eco-genetic approach to modeling the dynamics of limited sex-structured populations with non-overlapping generations within mathematical biology. We propose a dynamic model of a population that is structured by sex, which allows for studying evolutionary processes. Note that adding sex structure to the model is important for understanding population abundance fluctuations, since sexual reproduction is one of the essential mechanisms of microevolutionary transformations and animal adaptation to the environment. The model assumes that density-dependent factors limit the survival of females and males at different intensities. In addition, we consider a population with genetic diversity in individuals’ reproductive potentials, controlled by one autosomal diallelic locus. Note that mathematical models rarely incorporate population genetic equations without simplifying Mendelian inheritance principles. The unpopularity of single-locus models is related to the ingrained notion of polygenic control of phenotypic traits. Increasing the number of loci that are considered significantly complicates the models and quickly deprives them of the possibility of adequately interpreting simulation results. However, using a single-locus diallelic inheritance model is quite justified; in particular, one of the principal lifecycle traits of the arctic fox is litter size, which is genetically determined by a single diallelic locus [72].

This paper aims to study the evolutionary dynamics of an ecologically limited population that is structured by sex with seasonal reproduction and non-overlapping generations. The reason for choosing populations with non-overlapping generations and a short life cycle with a limited breeding period is the occurrence of significant fluctuations in the abundance of such species [73,74,75,76,77,78]. An example of such a species is the larch bud moth, for which genetic polymorphism is one of the main factors regulating its abundance [79,80]. At an excessively high density, the genetic structure of the insect population changes, accompanied by the emergence of morphological forms that differ in several traits. In particular, dark-colored ecotypes are more sensitive to food quality, and their predominance during defoliation accelerates the completion of the population abundance peak phase [79]. Additionally, this species is affected by ecological limitations due to decreasing food availability.

The present study continues a series of papers that is devoted to the dynamics of a limited population with sex structure [81,82,83,84], focusing on ecological modeling. This study considers changes in the reproductive potential of a population during evolution under the influence of natural selection. Consequently, the proposed model combines an ecological model of the dynamics of a limited population with non-overlapping generations represented by females and males with a microevolutionary model of the dynamics of the population’s genetic structure. We analyze the bistability of monomorphic solutions. We deliberately make minimal assumptions about the ecological structure of the population. A relatively simple model allows us to consider the influence of genotype reproductive potentials, secondary sex ratio (proportion of newborn females), and intraspecific competition on the dynamics of a sex-structured population and its genetic composition.

2. Model Assumptions

Let us consider a population with seasonal breeding, consisting of females and males. The parental generation dies before the new breeding season starts, ensuring non-overlapping population generations. Some species of plants, insects, fish, zooplankton, and birds have non-overlapping generations.

We consider the time step in modeling n to be the interval between breeding seasons. We assume that the number of females determines that of newborns, and the survival rates of females and males differ due to intraspecific competition and depend on population abundance.

In addition, we assume that there is genetically determined variety among individuals in the population, differing in reproductive potential (the maximum of their possible fecundity). To understand the basic outcomes of natural selection in a considered population, we can study a simple case where a single diallelic locus with alleles A and a encodes fitness. We assume that genes located on non-sex chromosomes (autosomes) control the reproductive potential, and there are three genotypic groups, AA, Aa, and aa, in the population, with different reproductive potentials that are equal to r_AA, r_Aa, and r_aa, respectively. Since the gene is autosomal, these three genotypic groups exist in both sexes for this trait.

Let us assume that the population is panmictic, i.e., that there is free mating between individuals with different genotypes, and Hardy–Weinberg equilibrium holds, linking genotype frequencies and allele frequencies [35]. In this case, to describe the dynamics of the population’s genetic structure, it is sufficient to consider the frequency dynamics of one of the alleles; for certainty, it will be the dynamics of allele A. It is easy to show that for a population with non-overlapping generations, after the first panmictic mating, the allele A frequencies in females and males of the offspring equalize, and their further dynamics do not differ (Appendix A).

As mentioned earlier, the ecological limitation of population growth is assumed by a decreasing female and male survival with an increasing total abundance [18]. We consider the linear decrease with different limitation (or competition) coefficients for females and males. Moreover, the limitation is suggested to not depend on the genotype, i.e., the survival of all female genotypic groups decreases with an increasing abundance according to the same law, and the same is true for the male genetic groups. The difference in male and female limitation coefficients does not lead to a difference in gene frequency dynamics in either sex of the populations. Then, one equation can describe the dynamics of allele A frequency in the population.

3. Model Description

The above assumptions are sufficient to obtain the following system of recurrent equations to describe the dynamics of female and male abundances, as well as allele A frequency in a population with non-overlapping generations and a genetic variety in individuals in terms of reproductive potential (Appendix A):

$$\left\{\begin{array}{c}F(n+1)=\mathsf{\delta }\overline{r}(n) F(n)(1-{\mathsf{\beta }}_1(F(n)+M(n)))\\ M(n+1)=(1-\mathsf{\delta })\overline{r}(n)F(n)(1-{\mathsf{\beta }}_2(F(n)+M(n)))\\ p_A(n+1)=p_A(n)(r_{AA}{p}_A(n)+r_{Aa}(1-p_A(n))/\overline{r}(n)\end{array}\right.,$$

(1)

where n is the breeding period number; F (F ≥ 0) and M (M ≥ 0) are female and male abundances, respectively; $p_A(n)$ ($0\le p_A\le 1$) is the frequency of allele A in the n-th generation; ${\mathsf{\beta }}_1$ (${\mathsf{\beta }}_1\ge 0$) and ${\mathsf{\beta }}_2$ (${\mathsf{\beta }}_2\ge 0$) are limitation coefficients describing the intensity of intrapopulation competition; and $\mathsf{\delta }$ ($0\le \mathsf{\delta }\le 1$) is the proportion of newborn females. Parameters $r_{AA}$, $r_{Aa}$, and $r_{aa}$ ($r_{AA},r_{Aa},r_{aa}\ge 0$) are the reproductive potentials of genotypes AA, Aa, and aa, respectively; $\overline{r}(n)=r_{AA}{(p_A(n))}^2+2r_{Aa}{p}_A(n)(1-p_A(n))+r_{aa}{(1-p_A(n))}^2$, $\overline{r}>0$ is the average reproductive potential of individuals in the population in the n-th breeding season.

After the transition to relative abundances ($F{\mathsf{\beta }}_2\to f$,$M{\mathsf{\beta }}_2\to m$), we obtain equations for the evolutionary dynamics of a population with two sexes in a simplified form:

$$\left\{\begin{array}{c}f(n+1)=\mathsf{\delta }\overline{r}(n) f(n)(1-\mathsf{\rho }(f(n)+m(n)))\\ m(n+1)=(1-\mathsf{\delta })\overline{r}(n)f(n)(1-(f(n)+m(n)))\\ p_A(n+1)=p_A(n)(r_{AA}{p}_A(n)+r_{Aa}(1-p_A(n))/\overline{r}(n)\end{array}\right.,$$

(2)

where $\mathsf{\rho }={\mathsf{\beta }}_1/{\mathsf{\beta }}_2$ ($\mathsf{\rho }\ge 0$) denotes a parameter that characterizes the relative contribution of mature individuals to the limitation of the population survival process.

The first two equations of system (2) describe the dynamics of female and male abundance when the average reproductive potential $\overline{r}(n)$ changes during microevolution. The third equation of system (2) corresponds to the dynamics of the A gene frequency, which determines reproductive potential, i.e., the microevolution process itself.

4. Fixed Points and Their Stability

In the third equation of system (2), the change in allele A frequency under the influence of natural selection does not depend on the first two equations. This means that the frequency of allele A (value p_A) in the model under consideration does not depend on the numbers of females and males. The change in the frequency of allele A, i.e., for one generation, equals

$$\Delta p_A=p_A(n+1)-p_A(n)=\frac{p_A(n)(r_{AA}{p}_A(n)+r_{Aa}(1-p_A(n))}{r_{AA}{(p_A(n))}^2+2r_{Aa}{p}_A(n)(1-p_A(n))+r_{aa}{(1-p_A(n))}^2}-p_A(n).$$

We find the equilibrium values of the frequency of allele A by equating $\Delta p_A=0$, i.e., $p_A(1-p_A)\frac{p_A(r_{AA}-2r_{Aa}+r_{aa})+r_{Aa}-r_{aa}}{(r_{AA}-2r_{Aa}+r_{aa})p_A^2(n)+2(r_{Aa}-r_{aa})p_A+r_{aa}}=0$.

It follows from the last equation that in system (2), three equilibrium genetic structures are possible, in which the equilibrium frequency of allele A takes the values of either ${\overline{p}}_A=0$, or ${\overline{p}}_A=1$, or ${\overline{p}}_A=p^{*}$, where $p^{*}=(r_{aa}-r_{Aa})/(r_{AA}-2r_{Aa}+r_{aa})$, $0<p^{*}<1$, which is the same as in the evolution model of a homogeneous population [38,66,85]. The first two equilibrium states are monomorphic and correspond to the complete displacement of one allele by another. The third equilibrium corresponds to the maintaining polymorphism in the population, i.e., the population has both alleles A and a, and in general, all three genotypes are present: AA, Aa, and aa.

Substituting stationary values $\overline{r}$ into the first two equations of system (2) gives the values of female and male population numbers. Let us present all the stationary states of system (2).

System (2) has three fixed points with zero numbers of females and males, of which two are monomorphic—$E_0(0, 0, 0)$ (by allele a) and $E_1(0, 0, 1)$ (by allele A), and one is polymorphic—$E_2(0, 0, p^{*})$. Fixed points $E_0$, $E_1$, and $E_2$ correspond to population extinction.

In addition, model (2) has three fixed points with a non-zero population size:

• Monomorphic fixed point $E_3(\overline{f}, \overline{m}, 0)$ with respect to allele a has the following coordinates:

  $$\begin{gathered} \overline{f}=(1-r_{aa}\mathsf{\delta })/\left(r_{aa}\left((\mathsf{\rho }-1)\mathsf{\delta }(r_{aa}(\mathsf{\delta }-1)-1)-1\right)\right), \\ \overline{m}=(1-\mathsf{\delta })(1+r_{aa}(\mathsf{\rho }-1)\mathsf{\delta })/(\mathsf{\delta }\mathsf{\rho } \overline{f}), {\overline{p}}_A=0, \end{gathered}$$

  (3)

  where $r_{aa}>1/\mathsf{\delta }$.
  In this case, females and males in the population are represented only by individuals with the aa genotype, while the AA and Aa genotypes are absent.
• Monomorphic fixed point $E_4(\overline{f}, \overline{m}, 1)$ with respect to allele A has the coordinates

  $$\begin{gathered} \overline{f}=(1-r_{AA}\mathsf{\delta })/\left(r_{AA}\left((\mathsf{\rho }-1)\mathsf{\delta }(r_{AA}(\mathsf{\delta }-1)-1)-1\right)\right), \\ \overline{m}=(1-\mathsf{\delta })(1+r_{AA}(\mathsf{\rho }-1)\mathsf{\delta })/(\mathsf{\delta }\mathsf{\rho } \overline{f}), {\overline{p}}_A=1, \end{gathered}$$

  (4)

  where $r_{AA}>1/\mathsf{\delta }$. The fixed point $E_4(\overline{f}, \overline{m}, 1)$ corresponds to the situation when the population consists of individuals from both sex groups with only the AA genotype.
• Polymorphic fixed point $E_5(\overline{f}, \overline{m}, p^{*})$ has the coordinates

  $$\begin{gathered} \overline{f}=-C(B\mathsf{\delta }-C)/(((\mathsf{\rho }-1)B{\mathsf{\delta }}^2-(B+C)(\mathsf{\rho }-1)\mathsf{\delta }-C)B), \\ \overline{m}=(1-\mathsf{\delta })((\mathsf{\rho }-1)B\mathsf{\delta }+C)C/(\mathsf{\delta }\mathsf{\rho } \overline{f}), {\overline{p}}_A=p^{*}, \end{gathered}$$

  (5)

  where $p^{*}=(r_{aa}-r_{Aa})/(r_{AA}-2r_{Aa}+r_{aa})$ and $B=r_{AA}{r}_{aa}-r_{Aa}^2$, $C=r_{AA}-2r_{Aa}+r_{aa}$,
  which has its meaning if the following conditions hold: $\overline{f}>0$, $\overline{m}>0$, and $0<p^{*}<1$. Therefore, equilibrium $E_5(\overline{f}, \overline{m}, p^{*})$ exists when $r_{Aa}>\max (r_{AA},r_{aa})$, $r_{AA}>(\mathsf{\delta }{r}_{Aa}^2-2r_{Aa}+r_{aa})/(r_{aa}\mathsf{\delta }-1)$ or $r_{Aa}<\min (r_{AA},r_{aa})$, and $r_{AA}>(\mathsf{\delta }{r}_{Aa}^2-2r_{Aa}+r_{aa})/(r_{aa}\mathsf{\delta }-1)$. In the case of polymorphism, females and males in the population are represented by three genotypes: AA, Aa, and aa.

To find the boundaries of the stability domains of the fixed points of system (2), we use standard methods [86] based on the characteristic polynomial ${\mathsf{\lambda }}^3+S{\mathsf{\lambda }}^2+H\mathsf{\lambda }+J=0$ for the Jacobian matrix of system (2). This approach allows us to specify hypersurfaces corresponding to different bifurcations of codimension 1: (1) transcritical (TC) $H=-J-1-S$ at $\mathsf{\lambda }=1$, (2) period doubling (PD) $H=J-1+S$ at $\mathsf{\lambda }=-1$, and (3) Neimark–Sacker (NS) $H=SJ-J^2+1$ at $\mathsf{\lambda }=e^{\pm i\varphi }$ and $\left|\mathsf{\lambda }\right|=1$ [86].

Table 1. The boundaries of the stability domains of the fixed points of model (2).

Fixed Points	The Boundaries of Stability Domains
	Transcritical bifurcation line (TC)
E0, E3	$T{C}_0^1$: $r_{aa}\mathsf{\delta }-1=0$; $T{C}_0^2$: $r_{aa}-r_{Aa}=0$
E1, E4	$T{C}_1^1$: $r_{AA}\mathsf{\delta }-1=0$; $T{C}_1^2$: $r_{AA}-r_{Aa}=0$
E2, E5	$T{C}_1^2$: $r_{AA}-r_{Aa}=0$; $T{C}_0^2$: $r_{aa}-r_{Aa}=0$; $T{C}_2^1$: $\mathsf{\delta }(r_{AA}{r}_{aa}-r_{Aa}^2)-(r_{AA}-2r_{Aa}+r_{aa})=0$
	Period doubling bifurcation line (PD)	Neimark–Sacker bifurcation line (NS)
E3	$P{D}_3$: $\mathsf{\rho }=\frac{(\mathsf{\delta }-1)(r_{aa}\mathsf{\delta }+3)(r_{aa}\mathsf{\delta }-1)}{\mathsf{\delta }(r_{aa}^2{\mathsf{\delta }}^2-(r_{aa}^2-2r_{aa})\mathsf{\delta }-r_{aa}-3)}$	$N{S}_3$: $\mathsf{\rho }=\frac{(\mathsf{\delta }-1){(r_{aa}\mathsf{\delta }-1)}^2}{\mathsf{\delta }(r_{aa}^2{\mathsf{\delta }}^2-(r_{aa}^2+2r_{aa})\mathsf{\delta }+2r_{aa}+1)}$
E4	$P{D}_4$: $\mathsf{\rho }=\frac{(\mathsf{\delta }-1)(r_{AA}\mathsf{\delta }+3)(r_{AA}\mathsf{\delta }-1)}{\mathsf{\delta }(r_{AA}^2{\mathsf{\delta }}^2-(r_{AA}^2-2r_{AA})\mathsf{\delta }-r_{AA}-3)}$	$N{S}_4$: $\mathsf{\rho }=\frac{(\mathsf{\delta }-1){(r_{AA}\mathsf{\delta }-1)}^2}{\mathsf{\delta }(r_{AA}^2{\mathsf{\delta }}^2-(r_{AA}^2+2r_{AA})\mathsf{\delta }+2r_{AA}+1)}$
E5	$P{D}_5$: $\mathsf{\rho }=\frac{(B\mathsf{\delta }+3C)(\mathsf{\delta }-1)(B\mathsf{\delta }-C)}{\mathsf{\delta }(B^2{\mathsf{\delta }}^2-(B-2C)B\mathsf{\delta }-(B+3C)C)}$, $B=r_{AA}{r}_{aa}-r_{Aa}^2$, $C=r_{AA}-2r_{Aa}+r_{aa}$	$N{S}_5$: $\mathsf{\rho }=\frac{(\mathsf{\delta }-1){(B\mathsf{\delta }-C)}^2}{\mathsf{\delta }(B^2{\mathsf{\delta }}^2-(B+2C)B\mathsf{\delta }+(2B+C)C)}$, $B=r_{AA}{r}_{aa}-r_{Aa}^2$, $C=r_{AA}-2r_{Aa}+r_{aa}$

TC, NS, and PD are lines of transcritical, Neimark–Sacker, and period doubling bifurcations, respectively. The lines of transcritical bifurcations for the following pairs of fixed points E₀ and E₃, E₁ and E₄, and E₂ and E₅ coincide. Subscripts 3–5 in lines NS and PD indicate the number of fixed points (E₃–E₅) that lose their stability.

presents the bifurcation lines bounding the stability domains of the fixed points of model (2).

Figure 1 shows the stability domains of monomorphic and polymorphic solutions of system (2), with zero and non-zero numbers of females and males on the parameter plane $(\mathsf{\rho }, r_{AA})$.

The presented parametric portraits demonstrate the possible dynamics of female and male abundances in a population, depending on the ratio of reproductive potentials of different genotypes and the intensity of ecological limitation (Figure 1). Note that the loss of stability of equilibrium solutions with non-zero abundance (i.e., E₃, E₄, and E₅) is realized both according to the Neimark–Sacker scenario (NS) and the period doubling scenario (PD) (Table 1, Figure 1). At the same time, the genetic composition will remain constant and correspond either to monomorphism or polymorphism, depending on the ratio of reproductive potentials of the heterozygote and homozygotes.

First, let us fix the parameters of system (2) so that the reproductive potential of the heterozygote r_Aa takes intermediate or maximum values, i.e., during evolution with an increasing reproductive potential of genotype AA (r_AA), overdominance is replaced by directional selection (Figure 1a–c). At $r_{Aa}>r_{aa}$ and $r_{Aa}<1/\mathsf{\delta }$, the stability domain of system (2) on the plane $(\mathsf{\rho }, r_{AA})$ consists of the stability regions of fixed points $E_2$ (at $r_{AA}<r_{Aa}$), $E_1$ (at $r_{Aa}<r_{AA}<1/\mathsf{\delta }$), and $E_4$ (at $r_{AA}>1/\mathsf{\delta }$), separated by bifurcation lines $T{C}_1^2$ and $T{C}_1^1$ (Figure 1a). At $r_{AA}<r_{Aa}$ and $r_{aa}<r_{Aa}$, there is overdominance, i.e., the population is polymorphic, but due to insufficient reproduction, including due to the sex ratio, the population becomes extinct. With a further increasing reproductive potential $r_{AA}$ at $r_{aa}<r_{Aa}<r_{AA}<1/\mathsf{\delta }$, the most fecund and, therefore, the most successful genotype is AA; it displaces all others; however, its level of reproduction and the current sex ratio also lead to population extinction in the long run. If $r_{AA}>1/\mathsf{\delta }$, then the reproductive potential of genotype AA reaches a level at which a monomorphic A population survives and successfully develops. At the same time, when crossing bifurcation boundary $N{S}_4$, an increase in the reproductive potential of the homozygote AA ($r_{AA}$) leads to a loss of stability of the solution $E_4$ according to the Neimark–Sacker scenario: the dynamics of female and male abundance switch to a quasiperiodic mode. When crossing the $P{D}_4$ boundary with increasing $r_{AA}$, the loss of stability of the fixed point $E_4$ occurs according to the period doubling scenario: stable oscillations of abundance arise, generated by a cascade of period doubling bifurcations. Moreover, the loss of stability of the solution $E_4$ according to the period doubling scenario occurs at smaller values of the ecological limitation intensity $\mathsf{\rho }$ and, according to the Neimark–Sacker scenario, at higher values of $\mathsf{\rho }$.

An increase in the reproductive potential of the heterozygote $r_{Aa}$ ($r_{Aa}>1/\mathsf{\delta }$) leads to the fact that the stability region of system (2) on the $(\mathsf{\rho }, r_{AA})$ plane is formed by the stability areas of solutions $E_2$ (at $r_{AA}\le (\mathsf{\delta }{r}_{Aa}^2-2r_{Aa}+r_{aa})/(r_{aa}\mathsf{\delta }-1)$), $E_5$ (at $(\mathsf{\delta }{r}_{Aa}^2-2r_{Aa}+r_{aa})/(r_{aa}\mathsf{\delta }-1)<r_{AA}<r_{Aa}$), and $E_4$ (at $r_{AA}>r_{Aa}$), separated by bifurcation lines $T{C}_2^1$ and $T{C}_1^2$ (Figure 1b). In this case, with the increasing reproductive potential of genotype AA during evolution, monomorphism replaces polymorphism. Moreover, at $r_{AA}\le (\mathsf{\delta }{r}_{Aa}^2-2r_{Aa}+r_{aa})/(r_{aa}\mathsf{\delta }-1)$, the polymorphic population becomes extinct due to insufficient reproduction.

Higher $r_{Aa}$ values expand the stability domain of the polymorphic solution $E_5$ (Figure 1c). At the same time, with increasing $r_{Aa}$ values, the stability region of the monomorphic fixed point $E_4$ narrows due to the shift of the transcritical bifurcation line (Figure 1a–c). The boundaries corresponding to the period doubling ($P{D}_4$) and Neimark–Sacker ($N{S}_4$) bifurcations remain unchanged. In turn, changing the reproductive potential values of the aa homozygote in the range of $0<r_{aa}<r_{Aa}$ does not affect the type and shape of the $E_4$ solution stability region (Figure 1).

With the reproductive potential of the heterozygote being smaller than that of both homozygotes, the increase in the reproductive potential of genotype AA leads to the fact that a “bistability trap” replaces directional selection. The second row of Figure 1 demonstrates the bistability of monomorphic solutions. The stability region of system (2) consists of the stability regions of solutions $E_0$, $E_1$, and $E_4$ (Figure 1d); $E_1$, $E_3$, and $E_4$ (Figure 1e); and $E_3$ and $E_4$ (Figure 1f).

The bistability of monomorphic fixed points $E_0$ and $E_1$ and $E_0$ and $E_4$ takes place when the stability domains of the monomorphic A allele fixed points with zero ($E_1$) and non-zero ($E_4$) abundances, separated by the line $r_{AA}=1/\mathsf{\delta }$ (line $T{C}_1^1$), overlap with the stability region of the monomorphic a allele solution with a zero population number $(E_0)$ (Figure 1d). If the stability domains of fixed points $E_1$ and $E_4$ intersect with the stability region of equilibrium $E_3$ with non-zero abundance, the bistability of $E_3$ with $E_1$ and $E_3$ with $E_4$ arises, respectively (Figure 1e). The stability domain of solution $E_3$ in the parameter plane $(\mathsf{\rho }, r_{AA})$ is a strip parallel to the ordinate axis (Figure 1e,f). One can see that with increasing values of the parameter $\mathsf{\rho }$, characterizing ecological limitation, solution $E_3$ can lose stability according to the period doubling scenario (via $P{D}_3$) and the Neimark–Sacker scenario ($N{S}_3$). Figure 1e,f show that the stability areas of fixed point $E_3$ cover those of $E_4$. One can see domains where both solutions $E_3$ and $E_4$ are simultaneously stable or unstable, or where one of them is attractive but the other is not. This indicates the bistability of monomorphic solutions when the equilibrium dynamics of the population size with the aa genotype can coexist with periodic or quasiperiodic fluctuations in the population with genotype AA or vice versa.

Note that the picture of dynamic behavior on the parameter plane $(\mathsf{\rho }, r_{aa})$ is similar to $(\mathsf{\rho }, r_{AA})$, since the solutions of system (2) are symmetric to $r_{AA}$ and $r_{aa}$.

5. Type of Selection and Dynamics of System (2)

The type of selection and, accordingly, the dynamics of the genetic composition in system (2) depend on the ratio of reproductive potentials of the heterozygote and homozygotes $r_{AA}$, $r_{Aa}$, and $r_{aa}$ (Figure 2). Let us consider this in more detail.

5.1. Increased Reproductive Potential of Heterozygotes

With $r_{Aa}>\max (r_{AA},r_{aa})$, the population maintains stable polymorphism at the equilibrium allele A frequency, i.e., ${\overline{p}}_A=p^{*}$. At the same time, the monomorphic solutions ${\overline{p}}_A=0$ and ${\overline{p}}_A=1$ exist but are unstable. In this case, there is selection favoring heterozygotes, known as a balancing selection (or heterozygote advantage) [87].

In system (2), there can be two polymorphic fixed points with zero ($E_2$) and non-zero ($E_5$) female and male abundances. The polymorphic point with a non-zero population abundance $E_5$ is stable if, in addition to the condition of a selective advantage of heterozygotes, the following ratio holds: $r_{Aa}>r_{Aa}^{*}=\left(1+\sqrt{(r_{AA}\mathsf{\delta }-1)(r_{aa}\mathsf{\delta }-1)}\right)/\mathsf{\delta }$ (region 6 in Figure 2). However, at $r_{Aa}<r_{Aa}^{*}$, the polymorphic fixed point with a zero population size $E_2$ becomes stable (region 1 in Figure 2).

At low reproductive potential values of homozygotes compared with heterozygotes, i.e., $r_{AA}<r_{Aa}$ and $r_{aa}<r_{Aa}$, from any polimorphic initial condition, the allele A frequency monotonically tends to the polymorphic equilibrium ${\overline{p}}_A=p^{*}$ with zero (at $r_{Aa}<r_{Aa}^{*}$, Figure 3a) or non-zero (at $r_{Aa}>r_{Aa}^{*}$, Figure 3b) female f and male m abundances. Note that at $r_{Aa}<r_{Aa}^{*}$, the population maintains its genetic variety, but the growth rate is such that it eventually goes extinct. Accordingly, the level of successful reproduction at which the population survives depends not only on the genotype fitness but also on the secondary sex ratio (δ).

Consider a situation where the initial allele A frequency in the population is close to 1 at $r_{aa}<r_{AA}<r_{Aa}$ (Figure 3c). This arises when genotype AA becomes predominant in a population with two sexes due to an external factor’s influence and the number of individuals of other genotypes is too small. However, over time, while maintaining the increased reproductive potential of heterozygous individuals in the population, stable polymorphism is established (Figure 3c). Consequently, the other genotypes differing from AA can appear in the next generation under favorable conditions. A similar situation arises when the initial A allele frequency is close to 0 and $r_{AA}<r_{aa}<r_{Aa}$, as shown in Figure 3d.

5.2. Intermediate Values of the Reproductive Potential of Heterozygotes

$0<r_{aa}<r_{Aa}<r_{AA}$ or $0<r_{AA}<r_{Aa}<r_{aa}$, corresponding to so-called driving selection, lead to the establishment of monomorphism with more fitted alleles [87].

At $0<r_{aa}<r_{Aa}<r_{AA}$, the selection results in allele A fixation with allele a displacement. In this case, two monomorphic equilibria of the population genetic structure exist; one of them, ${\overline{p}}_A=0$, is unstable, and the other, ${\overline{p}}_A=1$, is stable.

The bifurcation line $r_{AA}=1/\mathsf{\delta }$ separates the stability domains of monomorphic fixed points with zero ($E_1$) and non-zero ($E_4$) densities of females and males with AA genotypes (regions 2 and 4 in Figure 2). With $r_{AA}<1/\mathsf{\delta }$, the monomorphic allele A fixed point with a zero population size is stable (region 2 in Figure 2). If $r_{AA}>1/\mathsf{\delta }$, then $E_4$ with non-zero density is stable (area 4 in Figure 2). For any p_A value in the range of $0<p_A<1$, the allele A frequency monotonically increases and tends to unity. Consequently, only individuals with the AA genotype survive in the population, while individuals with the aa and Aa genotypes become extinct. The population size tends to either zero with $r_{AA}<1/\mathsf{\delta }$ (Figure 4a) or a non-zero value with $r_{AA}>1/\mathsf{\delta }$ (Figure 4b).

With $0<r_{AA}<r_{Aa}<r_{aa}$, allele a is fixed, and allele A is eliminated for any polymorphic initial condition. In the case of $r_{aa}<1/\mathsf{\delta }$, the point $E_0$, corresponding to the zero monomorphic fixed point with respect to allele a, is attractive (region 3 in Figure 2). If $r_{aa}>1/\mathsf{\delta }$, then point $E_3$ is stable, which corresponds to a monomorphic allele a population with non-zero size (area 5 in Figure 2). Consequently, at $0<r_{AA}<r_{Aa}<r_{aa}$, for any non-zero initial condition, the frequency of allele A monotonically decreases to 0, and the population size tends to either zero with $r_{aa}<1/\mathsf{\delta }$ (Figure 4c) or a non-zero value with $r_{aa}>1/\mathsf{\delta }$ (Figure 4d).

5.3. Reduced Reproductive Potential of Heterozygotes

$r_{Aa}<\min (r_{AA},r_{aa})$ leads to the bistability of monomorphic states when two extreme genetic structures in the population are stable. Disruptive selection, also known as destructive selection, promotes the survival of extreme genotypes and the elimination of intermediate ones from generation to generation [87].

With a disruptive type of selection, a non-trivial polymorphic fixed point ${\overline{p}}_A=p^{*}$ exists, but it is unstable. There are two stable states in which the allele frequency is either ${\overline{p}}_A=0$ or ${\overline{p}}_A=1$. Which fixed point is attractive depends on the initial genetic structure of the population. If the allele A frequency is smaller than $p^{*}$ in the initial generation, then it drops to 0 ($p_A\to 0$). If the initial value $p_A(0)$ is in the range of $p^{*}<p_A(0)<1$, then the allele A frequency increases to 1 ($p_A\to 1$). Consequently, with a reduced reproductive potential in heterozygotes, the result of selection is determined by the values of the reproductive potentials of homozygotes and heterozygotes and by the initial genetic structure of populations.

System (2) reveals four variants of bistability for fixed points.

Variant 1: The bistability of fixed points $E_0$ and $E_1$, corresponding to monomorphic populations with aa and AA genotypes, respectively, with zero group sizes of females and males. The simultaneous coexistence of these stable fixed points occurs when the following conditions hold: $0<r_{Aa}<r_{AA}\le r_{aa}<1/\mathsf{\delta }$ or $0<r_{Aa}<r_{aa}\le r_{AA}<1/\mathsf{\delta }$ for system (2) parameters (area 7 in Figure 2). Consequently, depending on the allele A frequency in the initial generation, selection leads to either allele A fixation with $p^{*}<p_A(0)<1$ or its displacement with $0<p_A(0)<p^{*}$. In both cases, the population size decreases until it becomes extinct.

Variant 2: The bistability of fixed point $E_0$, corresponding to a zero monomorphic aa population and $E_4$ with an AA genotype population of non-zero size. The bistability of these solutions exists at $0<r_{Aa}<r_{aa}<1/\mathsf{\delta }<r_{AA}$ (region 8 in Figure 2a–c). For example, Figure 5 shows a trajectory and surface $p_A(f,m)=p^{*}$ in the phase space of system (2). The surface $p_A(f,m)=p^{*}$, denoted as $p_A=p^{*}$ in Figure 5a, is self-mapped, because all trajectories whose initial points are on this surface belong to it. Obviously, polymorphic fixed points $E_2(0, 0, p^{*})$ and $E_5(\overline{f}, \overline{m}, p^{*})$, where $0<p^{*}<1$, belong to this surface.

With the reduced reproductive potential of heterozygotes, $p_A(f,m)=p^{*}$ is a separatrix surface dividing the phase space of system (2) into two attraction basins of stable monomorphic fixed points. If the initial frequency of allele A $p_A(0)$ is in the range of $p^{*}<p_A(0)<1$ (point $B_1$ in Figure 5a), then all trajectories move away from this surface and tend to the monomorphic allele A fixed point with a non-zero population size (Figure 5a,b). At $0<p_A(0)<p^{*}$ (the initial point $B_2$ in Figure 5a), the trajectories tend to the monomorphic allele a solution with a zero population size (Figure 5a,c). The allele A frequency monotonically changes and corresponds to logistic growth (Figure 5b) or decline (Figure 5c).

In the case of the presence of unstable fixed points $E_1$ and $E_2$, the transition process to one of the monomorphic states $E_4$ or $E_0$ changes if the allele A frequency in the initial generation is close to the polymorphic equilibrium value $p^{*}$.

When the value $p_A(0)$ is within the range of $p^{*}<p_A(0)<1$ and close to the separatrix surface $p_A(f,m)=p^{*}$, for example, the initial point $B_3$ in Figure 5a, the system (2) trajectory, initially approaches a polymorphic fixed point $E_2$ and then tends to a zero monomorphic solution $E_1$ and subsequently achieves a monomorphic state $E_4$ with a non-zero population size (Figure 5a,d). Such dynamic behavior is the so-called “bottleneck effect”, when a catastrophic decrease in population size is caused by various reasons and, following its recovery, is accompanied by a reduction in genetic variety [53,56]. Figure 5d demonstrates that initially, the population is polymorphic; over time, its size decreases to a very small number of individuals. Because $r_{AA}>r_{aa}>r_{Aa}$, the genotype AA with the highest fitness fixes due to driving selection [85]. As a result, genetic variety diminishes, and the population will present only individuals with the genotype AA. Over time, the population size increases again, but genetic variety is not restored. As depicted in Figure 5d, due to passing through a population bottleneck, the population loses genotypes Aa and aa, which have lower reproductive potential and, in this case, result in extinction in the future. This type of system behavior can be considered and interpreted as an effect of evolutionary rescue. A U-shaped curve of population size over time characterizes the process of evolutionary rescue. Initially, the population size decreases due to poor adaptation to environmental conditions, and fitness increases as adaptive alleles spread in the population [56,58].

If the allele A frequency in the initial generation is close to $p_A(f,m)=p^{*}$ but smaller than the value $p^{*}$ (the initial point $B_4$ in Figure 5a, $0<p_A(0)<p^{*}$), then all trajectories moving away from this surface tend to allele a monomorphic solution $E_0$ with a zero population size (Figure 5a,e). This transition is initially accompanied by a reduction in population size but without a loss of genetic variety. Over time, however, genetic impoverishment occurs in the population, resulting in the survival of only homozygous aa individuals. Such processes are critical for the population’s development, and because the reproductive potential of homozygote aa is insufficient for survival, its value is in the range $r_{aa}<1/\mathsf{\delta }<r_{AA}$ and leads to population extinction.

Variant 3: The bistability of fixed points $E_1$ and $E_3$ when a monomorphic AA stable state with a zero population abundance coexists with a monomorphic aa solution with a non-zero population size. This bistability exists if the condition $0<r_{Aa}<r_{AA}<1/\mathsf{\delta }<r_{aa}$ holds (region 9 in Figure 2a,e,f).

In the phase space of system (2), the attraction basins of fixed points $E_1$ and $E_3$ are bounded by surfaces $p_A=1$, $p_A(f,m)=p^{*}$, and $p_A=0$ (Figure 6).

If the frequency of allele A in the initial generation $p_A(0)$ is in the range of $0<p_A(0)<p^{*}$, such as point $B_1$ in Figure 6a, then all trajectories tend to the monomorphic aa solution with a non-zero population size $E_3$ (Figure 6a,b). With $p_A(0)$ from the range of $p^{*}<p_A(0)<1$, for example, the starting point $B_2$ in Figure 6a, which is the monomorphic AA fixed point with zero abundance $E_1$, is attractive (Figure 6a,c).

Similar to the previous case, if the initial allele A frequency is close to the value $p^{*}$ for the polymorphic fixed point, then the presence of unstable solutions $E_0$ and $E_2$ leads to non-monotonic transition processes in the system dynamics.

Figure 6a shows an example with an initial point $B_3$ at $p_A(0)=p^{*}+{10}^{-6}$ when the initial allele A frequency $p_A(0)$ is slightly higher than the polymorphic equilibrium frequency $p^{*}$. We can see that at the beginning, $p_A$ tends to a value that is close to the fixed point $E_2$ while maintaining polymorphism (Figure 6d). Then, during evolution, allele A accumulates and displaces allele a, i.e., the population becomes monomorphic in allele A. However, the reproductive potential of AA genotype individuals is insufficient for population survival, since $r_{AA}<1/\mathsf{\delta }<r_{aa}$ (Figure 6a,d).

With the allele A frequency in the initial generation $p_A(0)$ being slightly smaller than $p^{*}$, as an example, for the starting point $B_4$ at $p_A(0)=p^{*}-{10}^{-6}$ in Figure 6a, system (2) trajectories tend firstly to solution $E_2$, then to point $E_0$, and only after that, they reach equilibrium $E_3$. Consequently, during evolution, allele a accumulates in the polymorphic population, which leads to allele A elimination (Figure 6a,e). At the same time, the numbers of females and males in the population initially decrease to a critical level, then slowly grow and reach equilibrium $E_3$.

Figure 6e illustrates a U-shaped curve of the changes in the group sizes of females and males over time. Initially, the population size decreases due to the low reproductive potential of individuals with genotypes Aa and AA. During evolution, the more adaptive allele a accumulates, which leads to population size growth caused by selection and the survival of homozygous individuals with the most suitable genotype aa. Here, as in the previous cases, a temporary sharp decline in population size can be considered as passing through a bottleneck. Evolutionary rescue of the population is achieved by increasing the frequency of the existing allele a, which was initially rare. Note that if allele a was not rare, then a catastrophic reduction in population size would not have been observed (Figure 6b).

Variant 4: The bistability of fixed points $E_3$ and $E_4$, corresponding to the coexistence of two monomorphic equilibria with non-zero group sizes of females and males. System (2) reveals this bistability if the conditions $0<r_{Aa}<r_{aa}<r_{AA}$, $r_{aa}>1/\mathsf{\delta }$ (domain 10 in Figure 2a,c,d), or $0<r_{Aa}<r_{AA}<r_{aa}$, $r_{AA}>1/\mathsf{\delta }$ (domain 10 in Figure 2a,d,f) hold.

Similar to the previous cases, in the system (2) phase space, the attraction basins of fixed points $E_3$ and $E_4$ are bounded by the surfaces $p_A=0$, $p_A(f,m)=p^{*}$, and $p_A=1$, as shown in Figure 7a.

As in the case of the previous variant, with the initial point lying above the separatrix surface $p_A(f,m)=p^{*}$ (point $B_1$ in Figure 7a), the population becomes monomorphic with the AA genotype during evolution (Figure 7b). If the starting point is below $p^{*}$, such as $B_2$ in Figure 7a, then the evolution outcome is a monomorphic aa population (Figure 7c). However, if the allele A frequency in the initial generation $p_A(0)$ is close to its value $p^{*}$ for the polymorphic solution, such as for example the initial points $B_3$ and $B_4$ in Figure 7a, then the population can maintain polymorphism for a long time during the transition process (Figure 7d,e). Here, we observe the fixation of one of the alleles and the elimination of the other, which results in a monomorphic population (Figure 7d,e).

Thus, in this case, the outcome of evolution depends on the initial frequencies of allele A at the initial stage of population development. With $p<p^{*}$, allele A will be displaced; at $p>p^{*}$, it will fix. Consequently, the current genetic structure of the population determines the outcome of evolution due to occurrences of the so-called “bistability trap”, when a more promising genotype may be displaced by one with less fitness [25,26].

Above, we have considered various types of bistability of stable monomorphic solutions of system (2). Note that at higher values of the reproductive potential of genotypes, monomorphic fixed points with non-zero numbers lose stability according to the period doubling bifurcation or Neimark–Sacker scenario (Figure 1). As a result, we can observe the bistability of trivial/non-trivial stable solutions with periodic or quasiperiodic dynamics.

6. Dynamics of the Genetic Composition of the Population with Changes in the Reproductive Potential of Genotypes

This section considers the dynamics of the genetic composition of the population and the group sizes of females and males with the changing reproductive potential of individuals of different genotypes during evolution. An increase in reproductive potential can be caused, for example, by mutations when alleles with higher fitness appear. Accordingly, we assume that in a hypothetical monomorphic aa population, as a result of mutation, a new adaptive allele A₀ with a higher reproductive potential arises, and its frequency gradually increases. Initially, the ratio of the reproductive potentials of homozygotes and heterozygotes is chosen so that system (2) trajectories converge to the monomorphic A₀A₀ solution with a zero population size (range I in Figure 8a). After 200 iterations of a monotonous increase in the allele A₀ frequency and nearly fixing it in the population, a new allele A₁ with a higher reproductive potential appears due to another mutation while maintaining a near-zero population size. During driving selection, the A₁ allele frequency in the population monotonically grows, and allele A₁ displaces the A₀ allele. At the same time, the population size slowly increases, which corresponds to the tendency of the trajectory to a non-zero stationary state (range II in Figure 8a).

The population size curves for females and males are U-shaped (intervals I and II in Figure 8a), which corresponds to the evolutionary rescue of the population due to the increase in the frequency of a new allele A₁ with a higher reproductive potential that appears due to a mutation.

Furthermore, when the “good” allele A₁ becomes almost fixed and its frequency is close to $p_{A_1}=0.99$, a new better allele A₂ with a higher reproductive potential of homozygote $r_{A_2{A}_2}$ appears. With the emergence of this superior allele A₂, the current equilibrium population sizes of females and males approach a higher level (interval III in Figure 8a). As allele A₂ becomes more prevalent, it replaces allele A₁, and a new allele A₃ emerges, which has even higher fitness (interval IV in Figure 8a). Thus, the mutations and driving selection lead to a series of transitions, resulting in the growth of the average reproductive potential of the population over time (Figure 8a).

Figure 8b illustrates the process of the emergence of “good” alleles with reproductive potential whose values do not result in population extinction. Similar to the previous case, the arising mutants (alleles) compared with the wild species are assumed to have adaptive superiority. At driving natural selection, new alleles with higher fitness become fixed. If individuals with a new allele are superior in fitness to those with the wild allele, the new allele frequency in the population will gradually increase (Figure 8b). This process corresponds to the mechanism of allele fixation [87]. If the emergence of new alleles continues, this leads to fluctuations in population size over time due to the increase in the average reproductive potential of genotypes and density-dependent survival regulation (Figure 8b).

Consequently, genotypes with a higher reproductive potential become gradually fixed in the population, which leads to an evolutionary increase in their average reproductive potential. This results in an increase in the stationary numbers of females and males, as well as in bifurcations of the stability loss for current population equilibrium with an oscillation emergence.

7. Discussion

This study proposed and investigated the evolutionary dynamics model of a limited population that is structured by sex. We assumed genetic variety in the population concerning reproductive potential, controlled by a single autosomal diallelic locus. For the proposed model, we found monomorphic and polymorphic fixed points with conditions for their existence and stability (Table 1). The system is shown to reveal periodic and quasiperiodic oscillations, caused by the occurrence of a period doubling bifurcation and a Neimark–Sacker one (Figure 1). The scenario of system stability loss depends on the intensity of ecological limitations. However, the evolutionary results are always the same: with increasing genotypes’ reproductive potentials $r_{AA}$, $r_{Aa}$, and $r_{aa}$, the population dynamics become more irregular and complex.

We classified regions with different dynamics of population size and genetic composition (Figure 2). The outcomes of selection by reproductive potential values (F-selection) in ecologically limited populations with sex or stage structure [20,21] are shown to have no difference from those of Fishers’ fitnesses in unlimited ones. As expected, the condition for maintaining a stable polymorphism in a population is $r_{Aa}>\max (r_{AA},r_{aa})$, which corresponds to the highest value of the heterozygote reproductive potential (Figure 3). When $r_{aa}<r_{Aa}<r_{AA}$ or $r_{AA}<r_{Aa}<r_{aa}$, intermediate values of heterozygote fitness lead to a monomorphism for alleles with greater reproductive potential (Figure 4). The level of successful reproduction at which a population with two sexes can survive and grow depends on the ratio of reproductive potentials of the genotypes and the secondary sex ratio $\mathsf{\delta }$, which determines the proportion of female newborns.

With the reduced reproductive potential of heterozygotes $r_{Aa}<\min (r_{AA},r_{aa})$, the proposed model reveals bistability (Figure 5, Figure 6 and Figure 7). This bistability corresponds to the coexistence of both monomorphic fixed points, when the initial sex ratio and allele frequencies determine the genotype that will be fixed in the population. This scenario is a so-called “bistability trap”, which means that a higher-fitness genotype cannot displace those with lower fitness. However, a drop in population size due to external environmental factors leads to random processes, known as “passage through a bottleneck”, that can result in fluctuations in the genetic composition and a “shift” of the population from one monomorphic state to another one with more fitness, for which population dynamics destabilize.

In contrast to previously studied eco-evolutionary population models with a stage structure [20,21], the proposed model can show several types of bistability due to considering sexual dimorphism and, as a result, the secondary sex ratio. Depending on the initial allele frequencies and the sex ratio, the population may either become extinct or survive with the fixation of one of the alleles, as different monomorphic fixed points may coexist (Figure 5 and Figure 6). For example, one fixed point has a zero population size, while the other has non-zero numbers of males and females. This means that some initial conditions can increase the frequency of lower-fitness alleles, leading to population extinction due to a decrease in the growth rate (Figure 5c,e and Figure 6c,d). There are also initial conditions that can increase the frequency of an allele with a higher reproductive potential, for which, depending on the values of specific demographic parameters, the population size may show an equilibrium tendency or fluctuations (Figure 5b,d and Figure 6b,e). In this case, the variation in the current genetic composition of the population due to the influence of unaccounted environmental factors can change the “initial conditions” and, thus, the direction of evolution. Consequently, some external factors may lead to the extinction of a successfully developing population or its gradual recovery after a significant decline in its number.

The model study shows that, in the case of bottlenecks in population size, its genetic variety decreases, and if an allele with a higher reproductive potential is fixed, then the other one is eliminated (Figure 5d and Figure 6e). As a result, the population recovers because the genotype is cleared due to the removal of the low-fitness allele.

The population recovery in the proposed system illustrates so-called “evolutionary rescue” [56,57,58], where a population facing the threat of extinction is capable of adapting by increasing the frequency of alleles with higher fitness to prevent extinction. Figure 5d and Figure 6e demonstrate the case of evolutionary rescue when the population recovers due to the accumulation of initially rare alleles with a higher reproductive potential. The example of the regeneration of the natural population of the northern elephant seal, Mirounga angustirostris, allows us to suggest that this population has survived without fundamental changes, despite significant genetic diversity loss due to passing through a bottleneck [88]. However, strategies for population conservation and management require careful consideration of the possible consequences of the application of the developed measures.

Theoretical studies of the dynamics of structured populations with density-dependent regulation have shown that an increase in birth rate and survival parameters leads to more complex population dynamics [81,82,83,89]. Considering a genetic structure in the model of an ecologically limited sex-structured population, similar to those with a stage structure, allows us to investigate the possibilities of an evolutionary transition from equilibrium to fluctuations when an increase in the fertility of individuals occurs during evolution under the influence of natural selection. We have shown that the appearance of new alleles with a higher reproductive potential in individuals, for example due to mutations, leads to natural selection of these alleles and an evolutionary increase in the average birth rate of the population (Figure 8). However, density-dependent survival regulation with the growth of the average reproductive potential destabilizes the dynamics of the size of the female and male groups. We have shown that another scenario of evolutionary rescue occurs when the population recovers due to new alleles with a higher reproductive potential whose appearance is caused, for example, by mutations (Figure 8a). Note that the possibility of evolutionary rescue [22,23] and fluctuation emergence in limited populations [24,25,26] due to fixation of the alleles with higher fitness, whose appearance is caused by mutations, was expected, since this was previously demonstrated in dynamic models of stage-structured populations.

8. Conclusions

This study showed that an increase in the average value of the reproductive potential in a population with density-dependent regulation of survival destabilizes the dynamics of abundance of females and males. In this case, the scenario of stability loss of equilibrium via the period doubling or Neimark–Sacker bifurcations depends on the intensity of self-regulation. The genetic composition of the population, namely, monomorphic genotype fixation or coexistence of all genotypes in polymorphism, is shown to be determined by the values of the reproductive potentials of heterozygotes and homozygotes, initial conditions, and a parameter describing the ratio of newborn females to males. In particular, we found that the increased reproductive potential of heterozygotes leads to a stable polymorphism within a population. The intermediate value of the reproductive potential of heterozygotes results in a monomorphism with an allele with greater fitness. The reduced reproductive potential of heterozygotes induces bistable dynamics when the different monomorphic fixed points with zero and non-zero population sizes coexist. Thus, depending on the values of the initial allele frequencies and the parameter describing the ratio of newborn females to males, the population either becomes extinct or survives by fixing one of the alleles during evolution. In this scenario, the polymorphic state is unstable and is part of a transitional process in the dynamics to achieve one of the monomorphic states. With the coexistence of two monomorphic equilibria with zero and non-zero population sizes, the model of a population with two sexes proposed in this study reveals the possibility of bottleneck effects and evolutionary rescue. With the reduced reproductive potential of heterozygotes, the influence of environmental factors that are not considered in the model can alter the current genetic composition of the population, which can change the direction of evolution. This scenario corresponds to a situation in which a genotype with less fitness may displace a more promising form, and these processes are accompanied by passage through a bottleneck and evolutionary rescue. As a result, external factor influence can lead to the extinction of a successful developing population or a gradual population recovery after a noticeable decline in its abundance. This model reveals two variants of evolutionary rescue: when the population recovers due to rare alleles with higher fitness in disruptive selection or the emergence of new alleles caused by mutations in driving selection.