Optimal Control of a Genotype-Structured Prey–Predator Model: Strategies for Ecological Rescue and Oscillatory Dynamics Restoration

Abstract

Evolutionary changes can significantly impact interactions among populations and disrupt ecosystems by driving extinctions or collapsing population oscillations, posing substantial challenges to biodiversity conservation. This study addresses the ecological rescue of a predator population threatened by a mutant prey population using the optimal control method. To study this, we study a model that incorporates a genotypically structured prey population comprising wild-type, heterozygous, and mutant prey types, as well as the predator population. We prove that this model has both local and global existence and uniqueness of solutions, ensuring the model’s robustness. Then, we applied the optimal control method, incorporating Pontryagin’s Maximum Principle, to introduce a control input into the model and minimize the mutant population, thereby stabilizing the ecosystem. We utilize a reproduction number and a control efficacy measure to numerically demonstrate that the undesired dynamics of the model can be controlled, leading to the suppression of the mutant and the restoration of the oscillatory dynamics of the system. These findings demonstrate the applicability of optimal control strategies and provide a mathematical framework for managing such ecological disruptions.

1. Introduction

Understanding how ecological interactions and evolutionary processes operate on comparable timescales and creating complex feedback loops helps us understand how the population dynamics are shaped in natural ecosystems [1,2]. Mounting empirical evidence demonstrates that evolutionary adaptations can occur rapidly, fundamentally altering the dynamics among species populations [3,4]. This temporal convergence of ecological and evolutionary processes necessitates mathematical frameworks that explicitly incorporate genetic variation and selection pressures into population dynamics.

Population dynamics are necessarily shaped by interactions among the constituent species, environmental factors, and external perturbations. Shifting ecological conditions generally accelerate the creation of stressed and imbalanced ecosystems. Unbalanced competition can lead to catastrophic consequences, resulting in drastic changes to the environmental structure. Such disruptions may lead to disastrous outcomes, including species extinctions and the collapse of ecological oscillations, which threaten biodiversity and the stability of ecosystems. Invasion dynamics [5,6], altered predator–prey systems [7,8,9,10] are some of the examples where this imbalance in interactions has been observed in the field and studied(modeled) extensively. Thus, management programs aimed at controlling species populations in imbalanced ecosystems [6] must be equipped with adaptive strategies and a proper theoretical understanding of complex population dynamics. Additionally, the effects of these management programs must be adequately analyzed in terms of costs, budgets, and performance measures [11,12]. There are extensive works on the control of dynamics involving these ecological dynamics alone (for a detailed and complete bibliography, we refer to [12]).

In this work, we envisage one such scenario where, in the initial stages of the evolutionary dynamics, a prey mutates in a predator–prey system, and further, the prey is dangerous to the predator on consumption [13]. The emergence of harmful prey that can kill predators upon consumption fundamentally disrupts the energy flow and stability of predator–prey systems. The loss of traditional oscillatory dynamics represents a significant transition that can also lead to predator extinction, particularly when the prey population moves toward higher numbers of the harmful genotype. The mathematical models constructed (see Equation (6) on p. 60 of [13]) by incorporating genetic structure and the context of a harmful prey into predator–prey models reveal how evolutionary responses can dramatically alter the structure of the dynamical system. These systems were also studied for their rich dynamical behaviors, such as persistence and extinction behaviors, which critically depend on the parameters and functional forms of the nonlinearities used in modeling [13,14,15,16,17,18,19].

The mathematical models discussed above have their origins in naturally observed phenomena, including predator mortality resulting from mutations in prey populations. Typically, as explained in [13,20], the above scenario of harmful prey-inducing mortality of a predator may represent the initial stage of a co-evolutionary cycle that results in drastically altered ecosystem dynamics. Predator mortality due to prey has been well studied; for an excellent review, we refer to the one on reptiles (see p. 277 in [21]). There is also precedent in predator response to surprises of the form of prey showing sudden drastic changes [22]. Studies have also shown that some prey learning behaviors may overlook harmful prey and ultimately lead to mortality [23]. The works mentioned above indicate that these kinds of altered prey–predator systems can be observed in natural systems and are not purely theoretical in character.

In this work, we present a minimal model of a prey population with genetic polymorphism characterized by three genotypes (AA, Aa, and aa) following random mating structure, logistic regulation terms, and Holling Type II functional responses [13,24], including mortality differences [25]. The mathematical modeling of these phenomena requires coupling population dynamics with genetic-based equations [13], creating a rich dynamical system where ecological and evolutionary timescales interact. Unlike traditional models, where predation reduces prey numbers while benefiting predators, the presence of lethal prey introduces a mortality risk to predators from their primary food source. The mathematical consequences are that the system can exhibit persistence and extinction phenomena depending on initial conditions and parameter values. We are primarily interested in the extinction phenomena exhibited by these models, as presented in [13].

When predator populations become endangered or critically low in number, the importance of maintaining the persistence of species populations becomes very important for ecosystem stability and species conservation. Small predator populations are inherently vulnerable to stochastic extinctions due to demographic noise, environmental fluctuations, and genetic bottlenecks. The presence of harmful prey exacerbates these risks by creating additional mortality pressure. Regulation of population by interaction-based regulations, rather than solely by intrinsic mortality regulation mechanisms, can be more functionally rich, particularly as it gives rise to trophic levels or cascades [26,27], thereby providing robustness and resilience to disturbances. Hence, the goal of this work is to maintain the predator population and also ensure oscillatory dynamics.

Genetic changes can disrupt the pre-existing ecological balance, e.g., the poisonous newt-snake system as given in works of [8,9]. If these disruptions result in the endangerment of species, it will require carefully designed conservation strategies. These conservation strategies include designing external interventions, inclusive of the costs associated with such measures. Hence, this problem can be efficiently and robustly studied under the framework of optimal control. It is well known that equilibrium situations predicted by modeling evolutionary dynamics with ecological components can be drastically different from their counterparts [13,14,15]. Thus, these aspects, if not taken care of while designing management policies, are likely to cause serious inefficiencies in the designs. Optimal control theory [28,29,30,31,32] broadly refers to the area of dynamical systems where the problem is to achieve some desired dynamic behavior by control action, which is to be determined through minimizing a prescribed cost functional [11,12,33]. We observe that optimal control problems for evolutionary models with ecological dynamics have not been well-studied [34,35]. In this work, we focus on setting up an optimal control problem for eco-evolutionary predator–prey dynamics. The fact that there is a need for multi-species models is made abundantly clear in (Page: 253 in [6]) and in (Page: 97 [4]). These observations provide the rationale for choosing the minimal model, which captures the aforementioned competitive dynamics of multi-species growth without making unrealistic assumptions.

Optimal control problems (using Pontryagin’s Maximum Principle) have been used extensively in many systems consisting of predator and prey type interactions. We cite some relevant works that demonstrate the need to incorporate genetics-driven dynamics into ecological dynamics. Control of the citrus whitefly, which is a significant pest, was done in the work of [36]. Another instance of control of an Invasive alien species (IAS) is given in [37]. Recently, optimal control models have been designed for sustainable ecosystem development [38], and also inform decisions regarding harvesting populations, as discussed in [39]. However, we also observe that all of the work mentioned above deals only with ecological dynamics without any evolutionary features; thus, there is a need to address optimal controls of such eco-evolutionary problems.

We formulate an optimal control problem with a cost function that penalizes the mutant population and control effort and regularizes the dynamics of other populations. Pontryagin’s Maximum Principle (PMP) is employed to derive necessary conditions for optimal control, yielding a two-point boundary value problem (TPBVP) involving state and costate dynamics [11]. Due to the system’s high nonlinearity, analytical solutions are infeasible, necessitating the use of numerical methods to compute the control trajectory.

Brief Overview and Objectives

Our model system consists of a prey population with three distinct genotypes, governed by a system of nonlinear ordinary differential equations (ODEs). Based on the above contexts, the objective of this work is to study a nonlinear predator–prey model that captures mutant-driven ecological disruption and derive an optimal control strategy using the Pontryagin principle to achieve ecological rescue.

Theoretically, we establish the well-posedness of the model by proving the local existence and uniqueness of solutions, ensuring the system’s mathematical validity. Additionally, we demonstrate global existence and boundedness under a parameter condition, confirming that solutions remain biologically realistic over time. Numerically, we solve the TPBVP using the CasADi framework [40]. Simulations compare uncontrolled and controlled dynamics: without control, the mutant population grows, causing predator extinction and oscillation collapse. However, with optimal control, the mutant population is suppressed, the predator is rescued, and oscillatory dynamics are restored.

The paper is organized as follows: Section 2 presents the model, the optimal control problem, and the numerical methods. Section 3 details the theoretical results (existence and uniqueness, PMP analysis, numerical simulations). Section 4 discusses implications and future directions, followed by conclusions in Section 5.

2. Model and Methods

2.1. Predator-Prey Model Involving 3 Genotypes

We extend the generic Lotka-Volterra predator–prey model by incorporating genetically driven dynamics in the prey population through a single-locus, symmetric fertility model with mortality differences among genotypes [25]. The prey population has been divided into three genotypes: homozygous wild-type (AA), heterozygous (Aa), and homozygous mutant (aa) [13].

The growth dynamics of the three prey genotypes: wild-type (AA) ($x_1$), heterozygous (Aa) ($x_2$), mutant (aa) ($x_3$), and predator (y) populations are governed by a system of nonlinear ordinary differential equations (ODEs). The system has genotype-specific reproduction and Holling Type II functional response predation terms. The dynamics are chosen such that they reflect the scenario where the mutant prey population is harmful to the predator. Also, the escape rate is more than that of the other prey genotypes. Thus, the mutant prey, if unchecked, drives the predator to extinction by altering predation dynamics, leading to a loss of oscillatory behavior critical to ecological resilience. To counteract this, an optimal control model is set up. The control model has a key assumption that individuals with the $aa$ genotype can be clearly distinguished from the other two genotypes.

We emphasize that the model is not species-specific. Predator-prey ecosystems represent a canonical example of consumer-resource dynamics where the resource exhibits intrinsic renewal capacity. Control of such systems can be rigorously analyzed within the proposed mathematical framework. We reiterate that the model we have analyzed is not a typical predator–prey model, as the model incorporates evolutionary dynamics in the form of genotype-based matings. The external control intervention $u\left(t\right)$ can be made to represent various types of biologically informed management strategies e.g., culling efforts targeting the mutant population [41] or genetic engineering involving mutant plants (as in [42]) or genetic drives as used in [43]. For a complete review of such control interventions, we refer to the works in [34,35].

The state vector $\mathbf{x}={[x_1,x_2,x_3,y]}^T$ evolves in a biologically constrained domain

$$D=\{(x_1,x_2,x_3,y)\in {\mathbb{R}}^4:x_1,x_2,x_3\ge 0,y\ge 0,x_1+x_2+x_3\le K\}$$

ensuring non-negative populations and a total prey population bounded by the carrying capacity K. The domain is biological context-based, as a population cannot be negative, and a zero total prey population is unfeasible naturally. For more mathematical details regarding this type of constraint, we refer to the works of [13,17,18].

The optimal control problem is thus set up as given by the following nonlinear coupled ODEs

$$\begin{array}{cc}\hfill f_1:={\displaystyle \frac{d{x}_1}{dt}}& ={\alpha }_{1}x\left[{\beta }_{11}{\left({\displaystyle \frac{x_1}{x}}\right)}^2+{\beta }_{12}{\displaystyle \frac{x_1{x}_2}{x^2}}+{\beta }_{22}{\displaystyle \frac{1}{4}}{\left({\displaystyle \frac{x_2}{x}}\right)}^2\right]-{\displaystyle \frac{{\alpha }_1}{K}}{x}_{1}x-{\displaystyle \frac{x_1}{x}}{\displaystyle \frac{m_{1}xy}{a+x}}-d_1{x}_1\hfill \end{array}$$

(1)

$$\begin{array}{ll}\hfill f_2:={\displaystyle \frac{d{x}_2}{dt}}& =2{\alpha }_{2}x\left[{\beta }_{13}{\displaystyle \frac{x_1{x}_3}{x^2}}+{\displaystyle \frac{1}{2}}{\beta }_{23}{\displaystyle \frac{x_2{x}_3}{x^2}}+{\beta }_{12}{\displaystyle \frac{1}{2}}{\displaystyle \frac{x_1{x}_2}{x^2}}+{\displaystyle \frac{1}{4}}{\beta }_{22}{\left({\displaystyle \frac{x_2}{x}}\right)}^2\right]-{\displaystyle \frac{{\alpha }_2}{K}}{x}_{2}x-{\displaystyle \frac{x_2}{x}}{\displaystyle \frac{m_{2}xy}{a+x}}-d_2{x}_2\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill f_3:={\displaystyle \frac{d{x}_3}{dt}}& ={\alpha }_{3}x\left[{\beta }_{33}{\left({\displaystyle \frac{x_3}{x}}\right)}^2+{\beta }_{32}{\displaystyle \frac{x_3{x}_2}{x^2}}+{\displaystyle \frac{1}{4}}{\beta }_{22}{\left({\displaystyle \frac{x_2}{x}}\right)}^2\right]-{\displaystyle \frac{{\alpha }_3}{K}}{x}_{3}x-{\displaystyle \frac{x_3}{x}}{\displaystyle \frac{m_{3}xy}{a+x}}-d_3{x}_3-u\left(t\right)x_3\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill f_4:={\displaystyle \frac{dy}{dt}}& =y\left[{\displaystyle \frac{m_1{x}_1+m_2{x}_2}{a+x}}\right]-sy-{\displaystyle \frac{m_3{x}_{3}y}{a+x}}\hfill \end{array}$$

(4)

where $x=x_1+x_2+x_3$ is the total prey population, and ${\alpha }_i,{\beta }_{ij},K,m_i,a,d_i,s>0$ are strictly positive parameters. All parameters are defined with their biological meanings and units in

Table 1. Summary of variables and parameters used in the ecological control model, including state variables, model parameters, cost terms, and numerical settings.

Symbol	Variable Name	Description
State and Control Variables
$\mathbf{x}={[x_1,x_2,x_3,y]}^T$	State vector	Wild-type (AA) ($x_1$), heterozygous (Aa) ($x_2$), mutant (aa) ($x_3$), and predator (y) population. $\mathbf{x}\in D\subset {\mathbb{R}}^4$, with $D=\{\mathbf{x}:x_i,y\ge 0,{\sum }_{i=1}^3{x}_i\ge ϵ\}$
$u\left(t\right)$	Control input	Suppression effort targeting mutants; $u\in [0,u_{max}]$
$\mathit{\lambda }$	Costate vector	Costates for Pontryagin’s Maximum Principle (PMP), $\mathit{\lambda }\in {\mathbb{R}}^4$ [12]
Model Parameters
${\alpha }_i$	Growth rates	Intrinsic growth for $x_i$, ${\alpha }_i>0$ [13]
${\beta }_{ij}$	Fertilities	Fertilities values of various genotype matings, ${\beta }_{ij}>0$ [13]
K	Carrying capacity	Population limit, $K>0$ [13]
$m_i$	Maximal Predation rates	Maximal value of the predation rate, $m_i>0$ [24]
a	Affinity constant	Prey density at which predation rate is half maximal, $a>0$ [24]
$d_i$	Mortality rates	Natural mortality depending on genotype, $d_i>0$ [25]
Cost Parameters
q	Mutant penalty	Weight on $x_3$ deviation, $q>0$
c	Control cost	Weight on effort, $c>0$
p	Terminal cost	Final $x_3$ penalty, $p>0$
$x_{3,\mathrm{target}}$	Target	Desired $x_3$ level

along with the references where more details can be found about the parameters. The control input $u\left(t\right)\in \mathbb{R}$ is bounded, $u\left(t\right)\in [0,u_{\max }]$, reflecting practical limits on suppression efforts.

The derivation of the equation follows general mass-action type mechanisms, i.e., gain and loss terms. The positive sign terms contribute to the gain, and the negative sign terms contribute to the loss. First, we describe the prey equations (${\displaystyle \frac{d{x}_i}{dt}}$). The terms in the square brackets in the case of $f_1,f_2,f_3$ comes from the various fertility (${\beta }_{ij}$) of matings of AA, Aa and aa type contributing to the growth of the respective populations (For a full table of the mating types we refer to the table in p. 20 in [44]). The loss terms are of three types: first, the term containing K is from the logistic carrying capacity terms, second, the terms containing the $d_i$’s are intrinsic mortality terms, and third, the terms containing $m_i$’s are the predation-induced functional response loss term. Next, we describe the equation of the predator denoted by $f_4$, i.e., the (${\displaystyle \frac{dy}{dt}}$). The gain term is due to the consumption of the two prey genotypes, which are not harmful to the predator. There are two loss terms: first, the term containing $m_3$, which represents the loss term due to the consumption of the dangerous prey, and second, the term containing s is the intrinsic mortality term.

The hypothesis in [13], where the uncontrolled version of this model was presented, was that if the relevant parameters are set equal, the model recovers the standard predator–prey equations. In (1)–(3), the quantities in square brackets can also have a probabilistic interpretation. The quantity ${\displaystyle \frac{x_1}{x}}$ can be thought of the probability of selecting a parent of type $x_1$ so, for example, ${\displaystyle \frac{x_1}{x}}·{\displaystyle \frac{x_1}{x}}$ is the probability of both parents being of type $x_1$ while ${\displaystyle \frac{x_1}{x}}·{\displaystyle \frac{x_2}{x}}$ represents the probability of parents of type $x_1$ and $x_2$, half of whose offspring will be of type $x_1$, and the parents may be selected in either order. The offspring produced by each mating pair combination are shown in the representative Figure 1. The arrows represent the flow of genes from parents to offspring.

To achieve ecological rescue and restore oscillatory behavior in the genotype-structured prey–predator system defined by Equations (1)–(4), we formulate an optimal control problem to minimize the impact of the harmful mutant prey population (aa, denoted $x_3$) while balancing the costs of efforts. The control input intervention $u\left(t\right)$ aims to prevent predator extinction and maintain ecological stability. We employ an optimal control framework to determine the optimal control strategy $u^{*}\left(t\right)$ that minimizes a cost functional subject to the nonlinear dynamics in Equations (1)–(4). Below, we present a general optimal control formulation [11,12,33], which we subsequently apply to our system.

2.2. Optimal Control Formulation and Numerical Methods

Consider a general nonlinear dynamical system

$$\dot{\mathbf{x}}\left(t\right)=f(\mathbf{x}\left(t\right),u\left(t\right)),\mathbf{x}\left(0\right)={\mathbf{x}}_0,$$

where $\mathbf{x}\left(t\right)\in {\mathbb{R}}^n$ is the state vector, $u\left(t\right)\in \mathcal{U}\subset {\mathbb{R}}^m$ is the control input, and $f:{\mathbb{R}}^n\times {\mathbb{R}}^m\to {\mathbb{R}}^n$ is a nonlinear function governing the system dynamics. In our model, $\mathbf{x}={[x_1,x_2,x_3,y]}^T$ represents the populations of wild-type (AA), heterozygous (Aa), mutant (aa), and predator, respectively, with $u\left(t\right)\in [0,u_{\max }]$ as the bounded control targeting the mutant population $x_3$. The objective is to minimize the cost functional

$$J\left(u\right)=\varphi \left(\mathbf{x}\left(T\right)\right)+{\int }_0^{T}L(\mathbf{x}\left(t\right),u\left(t\right))dt,$$

where $\varphi :{\mathbb{R}}^n\to \mathbb{R}$ is the terminal cost, penalizing deviations from desired population levels at the final time $T>0$, and $L:{\mathbb{R}}^n\times {\mathbb{R}}^m\to \mathbb{R}$ is the running cost, balancing ecological goals with the cost of applying control.

To incorporate the system dynamics as constraints, we define the augmented objective function using Lagrange multipliers $\mathit{\lambda }\left(t\right)\in {\mathbb{R}}^n$, known as costate variables

$$J_a(u,\mathbf{x},\mathit{\lambda })=\varphi \left(\mathbf{x}\left(T\right)\right)+{\int }_0^T\left[L(\mathbf{x},u)+{\mathit{\lambda }}^T(f(\mathbf{x},u)-\dot{\mathbf{x}})\right]dt.$$

This formulation transforms the constrained optimization problem into a calculus of variations problem. The Hamiltonian, which combines the running cost and the dynamics, is defined as

$$\mathcal{H}(\mathbf{x},u,\mathit{\lambda })=L(\mathbf{x},u)+{\mathit{\lambda }}^{T}f(\mathbf{x},u).$$

Pontryagin’s Maximum Principle provides the necessary conditions for optimality, which we apply to our system to find the optimal control $u^{*}\left(t\right)$. These conditions are

• State Dynamics

  $${\dot{\mathbf{x}}}^{*}\left(t\right)={\displaystyle \frac{\partial \mathcal{H}}{\partial \mathit{\lambda }}}=f({\mathbf{x}}^{*}\left(t\right),u^{*}\left(t\right)),{\mathbf{x}}^{*}\left(0\right)={\mathbf{x}}_0.$$
  This condition ensures that the optimal state trajectory ${\mathbf{x}}^{*}\left(t\right)={[x_1^{*}\left(t\right),x_2^{*}\left(t\right),x_3^{*}\left(t\right),y^{*}\left(t\right)]}^T$ follows the system dynamics given by Equations (1)–(4) under the optimal control $u^{*}\left(t\right)$. For our model, this means the populations of wild-type, heterozygous, mutant, and predator evolve according to the nonlinear ODEs, with $u^{*}\left(t\right)$ reducing the mutant population $x_3$. The initial condition ${\mathbf{x}}_0={[x_1\left(0\right),x_2\left(0\right),x_3\left(0\right),y\left(0\right)]}^T$ specifies the starting populations.
• Costate Dynamics

  $$\dot{\mathit{\lambda }}\left(t\right)=-{\displaystyle \frac{\partial \mathcal{H}}{\partial \mathbf{x}}}({\mathbf{x}}^{*}\left(t\right),u^{*}\left(t\right),\mathit{\lambda }\left(t\right)),\mathit{\lambda }\left(T\right)={\displaystyle \frac{\partial \varphi }{\partial \mathbf{x}}}\left({\mathbf{x}}^{*}\left(T\right)\right).$$
  The costate variables $\mathit{\lambda }\left(t\right)={[{\lambda }_1\left(t\right),{\lambda }_2\left(t\right),{\lambda }_3\left(t\right),{\lambda }_4\left(t\right)]}^T$ represent the sensitivity of the cost functional to changes in the state variables. This equation describes how the costates evolve over time, reflecting the trade-offs between population levels and control costs. The terminal condition at $t=T$ ensures that the costates align with the terminal cost $\varphi$, which in our case may penalize low predator (y) or wild-type ($x_1$) populations, encouraging ecological balance.
• Minimization Condition

  $$u^{*}\left(t\right)=arg\underset{u\in \mathcal{U}}{min}\mathcal{H}({\mathbf{x}}^{*}\left(t\right),u,{\mathit{\lambda }}^{*}\left(t\right)).$$
  This condition requires that the optimal control $u^{*}\left(t\right)$ minimizes the Hamiltonian at each time t, given the optimal state ${\mathbf{x}}^{*}\left(t\right)$ and costate ${\mathit{\lambda }}^{*}\left(t\right)$. In our model, since $u\left(t\right)\in [0,u_{\max }]$, we select the control that minimizes the Hamiltonian, balancing the reduction of the mutant population $x_3$ (via the term $-u\left(t\right)x_3$ in Equation (3)) against the control cost in $L(\mathbf{x},u)$. This ensures that suppression efforts are effective yet practical.

The optimal control $u^{*}\left(t\right)$ is determined by solving a two-point boundary value problem (TPBVP) derived using Pontryagin’s Maximum Principle [11], which is well-posed under the Lipschitz continuity of the system dynamics (1)–(4) and the bounded control set $u\left(t\right)\in [0,u_{\max }]$ as proved in Section 3. The TPBVP is solved numerically using the shooting method [11,45], implemented within the CasADi [40] framework for efficient computation and optimization. The time horizon $[0,T]$ is discretized into N intervals with step size $\Delta t=T/N$, defining state $\mathbf{x}\left(t_k\right)\in {\mathbb{R}}^n$ and control $u\left(t_k\right)\in \mathcal{U}\subset {\mathbb{R}}^m$ at grid points $t_k=k\Delta t$, $k=0,\dots ,N$. The cost functional $J\left(u\right)=\varphi \left(\mathbf{x}\left(T\right)\right)+{\int }_0^{T}L(\mathbf{x},u)dt$ is approximated via numerical quadrature. The dynamics $\dot{\mathbf{x}}=f(\mathbf{x},u)$ are enforced as equality constraints using the CVODES integrator from SUNDIALS [46]. The resulting nonlinear programming problem (NLP) is formulated in CasADi and solved using the IPOPT [47] solver with the MUMPS linear solver and tolerance ${10}^{-2}$, respecting control constraints $u\in \mathcal{U}$ and initial condition $\mathbf{x}\left(0\right)={\mathbf{x}}_0$. This yields the optimal control $u^{*}\left(t\right)$.

2.3. Reproduction Metrics and Control Efficacy

To assess the growth potential of the mutant prey population ($x_3$, aa genotype) and the effectiveness of the control strategy in an ecological context, we derive two quantities: first, the basic reproduction number ($R_0^{x_3}$) following in spirit the works in [48,49,50] and second, a composite control efficacy metric based on the system dynamics described by Equations (1), (2), (3), and (4) for $x_1$, $x_2$, $x_3$, and y, respectively.

Reproduction Numbers: The basic reproduction number ($R_0^{x_3}$) represents the average number of offspring produced by a single $x_3$ individual (aa genotype) over its lifetime in a low-density environment, free from predation, competition, or control [48]. Ecologically, $R_0^{x_3}$ quantifies the intrinsic growth potential of the mutant population in ideal conditions, such as when it first emerges in a prey community, i.e., coefficients of terms linear in $x_3$. From the dynamics of $x_3$ in (3), the gain term simplifies to ${\alpha }_3{\beta }_{33}{x}_3$, and the loss term is $d_3{x}_3$. Thus, without the control term, we get

$$R_0^{x_3}={\displaystyle \frac{{\alpha }_3{\beta }_{33}}{d_3}}.$$

With control ($u\left(t\right)\ne 0$), the loss rate increases; thus the above equation becomes

$$R_0^{x_3}={\displaystyle \frac{{\alpha }_3{\beta }_{33}}{d_3+u\left(t\right)}}.$$

Biologically, a high $R_0^{x_3}$ indicates the mutant’s potential to proliferate rapidly if unchecked, posing a risk to ecosystem balance, while a reduced $R_0^{x_3}$ under control reflects diminished reproductive capacity.

The effective reproduction number ($R_e^{x_3}$) measures the actual number of offspring per $x_3$ individual under current ecological conditions, including competition with other genotypes ($x_1$, $x_2$), predation by y, and control efforts. Ecologically, $R_e^{x_3}<<1$ is critical for preventing its dominance in the prey community. From (3) we have the gross gain and loss terms as

$${\displaystyle \frac{d{x}_3}{dt}}={\alpha }_{3}x\left[{\beta }_{33}{\displaystyle \frac{x_3^2}{x}}+{\beta }_{32}{\displaystyle \frac{x_3{x}_2}{x}}+{\displaystyle \frac{1}{4}}{\beta }_{22}{\displaystyle \frac{x_2^2}{x}}\right]-{\displaystyle \frac{{\alpha }_3}{K}}{x}_{3}x-{\displaystyle \frac{x_3}{x}}{\displaystyle \frac{m_{3}xy}{a+x}}-d_3{x}_3-u\left(t\right)x_3$$

The growth rate term, representing births from mating interactions, is

$$\mathrm{Growth}\mathrm{terms}={\alpha }_3\left({\beta }_{33}{\displaystyle \frac{x_3^2}{x}}+{\beta }_{32}{\displaystyle \frac{x_2{x}_3}{x}}+0.25{\beta }_{22}{\displaystyle \frac{x_2^2}{x}}\right)$$

and the loss rate, including natural mortality, control, predation, and density-dependent competition, is

$$\mathrm{Loss}\mathrm{term}=d_3+u\left(t\right)+{\displaystyle \frac{m_{3}y}{a+x}}+{\displaystyle \frac{{\alpha }_3}{K}}x$$

Thus, the effective reproduction rate becomes

$$R_e^{x_3}={\displaystyle \frac{{\alpha }_3\left({\beta }_{33}\frac{x_3}{x}+{\beta }_{32}\frac{x_2}{x}+0.25{\beta }_{22}\frac{x_2^2}{x_{3}x}\right)}{d_3+u\left(t\right)+\frac{m_{3}y}{a+x}+\frac{{\alpha }_3}{K}x}}.$$

Biologically, $R_e^{x_3}$ reflects the mutant’s ability to persist in a dynamic ecosystem, where predation and control can suppress its spread, maintaining biodiversity and ecosystem stability.

Control Efficacy: The control efficacy is a time-dependent composite metric designed to evaluate the ecological and practical success of the control strategy in achieving three objectives: (1) suppressing the mutant prey population ($x_3$) below a target threshold ($x_3^{\mathrm{target}}=0.001$), (2) stabilizing the predator population (y) near its target ($y^{\mathrm{target}}=0.21$) to ensure ecological balance, and (3) minimizing control effort ($u\left(t\right)$) to reduce intervention costs and environmental impact. Ecologically, this metric balances the need to control a potentially invasive mutant genotype, which could disrupt prey community dynamics, with the preservation of a functional predator–prey relationship. Biologically, suppressing $x_3$ prevents the mutant from outcompeting other genotypes ($x_1$, $x_2$), while stabilizing y supports predation as a natural control mechanism. The efficacy at time t is defined as

$$\mathrm{Efficacy}\left(t\right)=max\left(0,min\left(100,w_1{E}_{x_3}\left(t\right)+w_2{E}_y\left(t\right)-w_3{E}_u\left(t\right)\right)\right),$$

where:

• $E_{x_3}\left(t\right)=\left\{\begin{array}{cc}100\left(1-{\displaystyle \frac{x_3\left(t\right)-x_3^{\mathrm{target}}}{x_3^{\mathrm{target}}}}\right)\hfill & \mathrm{if}{x}_3\left(t\right)>x_3^{\mathrm{target}},\hfill \\ 100\hfill & \mathrm{otherwise},\hfill \end{array}\right.$ measures the percentage achievement of $x_3$ suppression, with 100% indicating $x_3\left(t\right)\le x_3^{\mathrm{target}}$.
• $E_y\left(t\right)=100\left(1-{\displaystyle \frac{|y\left(t\right)-y^{\mathrm{target}}|}{y^{\mathrm{target}}}}\right),$ quantifies how closely $y\left(t\right)$ matches the target, with 100% when $y\left(t\right)=y^{\mathrm{target}}$.
• $E_u\left(t\right)=100{\displaystyle \frac{\left|u\right(t\left)\right|}{u_{\max }}},$ penalizes control effort relative to a maximum input ($u_{\max }=10$), reflecting resource or ecological costs.

Weights $w_1=0.8$, $w_2=0.1$, and $w_3=0.1$ prioritize the suppression of $x_3$, reflecting its ecological urgency, while considering predator stability and control costs. The efficacy is clipped to $[0,100]$ to ensure a percentage scale. Ecologically, a high efficacy indicates successful management of the mutant population without destabilizing the predator–prey system or incurring excessive control costs, promoting sustainable ecosystem management.

3. Results

This section presents the theoretical results for the controlled population dynamics system, establishing the foundation for optimal control strategies to suppress the harmful mutant population ($x_3$) while maintaining ecological balance. We first prove the existence and uniqueness of solutions to ensure the system’s well-posedness. Then, we apply Pontryagin’s Maximum Principle (PMP) to derive necessary conditions for the optimal control, characterizing the suppression strategy.

3.1. Existence, Uniqueness, and Boundedness of Solutions

In this part, we prove two results: (i) We establish the existence and uniqueness, and (ii) global boundedness, implying the global existence of solutions to the dynamical system. For the proofs, we have followed the method given in (Section 6.2, pg: 149–150) of [51] and (Section 2.2, pg: 70 of [52]) and the methods outlined in [15,18,19].

Let $D=\{(x_1,x_2,x_3,y)\in {\mathbb{R}}^4:x_1,x_2,x_3,y\ge 0,0<x_1+x_2+x_3\le K\}$ denote the biologically feasible domain, where individual prey populations can be zero but not simultaneously, and the total prey population $x=x_1+x_2+x_3$ is bounded by the carrying capacity K. Let $U=[0,u_{max}]$ denote the admissible control set.

We first establish local existence and uniqueness of solutions.

Theorem 1.

Let D be as defined above, and let $U=[0,u_{max}]$. For any initial condition $\mathbf{x}\left(0\right)={\mathbf{x}}_0\in D$ and control $u\left(t\right)\in L^{\infty }[0,T]$, there exists a $\delta >0$ such that the system (1)–(4) has a unique solution $\mathbf{x}\left(t\right)\in C^1([0,\delta ];D)$ satisfying $\mathbf{x}\left(0\right)={\mathbf{x}}_0$.

Proof. 

Define $f(\mathbf{x},u)={[f_1\left(\mathbf{x}\right),f_2\left(\mathbf{x}\right),f_3(\mathbf{x},u),f_4\left(\mathbf{x}\right)]}^T$, where $f_1,f_2,f_3,f_4$ are the right-hand sides of (1)–(4) and given admissible control $u\left(t\right)\in L^{\infty }[0,T]$ is measurable and bounded. We verify that $f(\mathbf{x},u)$ is continuous in $\mathbf{x}\in D$ and $u\in U$, and satisfies Lipschitz continuity in $\mathbf{x}$, that is, for each $u\in U$, there exists $L>0$ such that $\parallel f({\mathbf{x}}_1,u)-f({\mathbf{x}}_2,u)\parallel \le L\parallel {\mathbf{x}}_1-{\mathbf{x}}_2\parallel$ for all ${\mathbf{x}}_1,{\mathbf{x}}_2\in D$.

(i.) Continuity Consider any ${\mathbf{x}}_0\in D$ and a neighborhood around it. Since $0<x_0=x_{1,0}+x_{2,0}+x_{3,0}\le K$, there exists a bounded subset

$$D_{\mathrm{loc}}=\{\mathbf{x}\in D:0\le x_1,x_2,x_3,y\le M,{\delta }_0\le x_1+x_2+x_3\le K\}$$

containing ${\mathbf{x}}_0$, where $M\ge K$, ${\delta }_0>0$, thus we get $x>0$. For $f_1$,

$$f_1\left(\mathbf{x}\right)={\alpha }_{1}x\left({\beta }_{11}{\displaystyle \frac{x_1^2}{x^2}}+{\beta }_{12}{\displaystyle \frac{x_1{x}_2}{x^2}}+{\beta }_{22}{\displaystyle \frac{x_2^2}{4x^2}}\right)-{\displaystyle \frac{{\alpha }_1}{K}}{x}_{1}x-{\displaystyle \frac{x_1{m}_{1}xy}{a+x}}-d_1{x}_1.$$

For $f_1$, the growth term involves fractions ${\displaystyle \frac{x_i{x}_j}{x^2}}$ which are continuous since $x\ge {\delta }_0>0$. The carrying capacity term $-{\displaystyle \frac{{\alpha }_1}{K}}{x}_{1}x$ is polynomial, the predation term $-{\displaystyle \frac{x_1}{x}}{\displaystyle \frac{m_{1}xy}{a+x}}$ is continuous as $a+x\ge a>0$ and $x\ge {\delta }_0$, and the death term $-d_1{x}_1$ is linear. At $y=0$, the predation term vanishes, preserving continuity. Similarly, $f_2$ and $f_3$ involve rational functions with denominators $x^2\ge {\delta }_0^2$ and $a+x\ge a$, ensuring continuity. The control term $-u{x}_3$ in $f_3$ is continuous since $\left|u\right|\le u_{max}$. For $f_4$

$$f_4\left(\mathbf{x}\right)=y\left({\displaystyle \frac{m_1{x}_1+m_2{x}_2}{a+x}}-s\right)-{\displaystyle \frac{m_3{x}_{3}y}{a+x}},$$

the fractions are continuous, and multiplication by y ensures continuity at $y=0$. Thus, $f(\mathbf{x},u)$ is continuous on $D_{loc}\times U$.

(ii.) Lipschitz Continuity We show that all the terms in the Jacobian ${\displaystyle \frac{\partial f}{\partial \mathbf{x}}}$ are bounded on $D_{loc}$.

For $f_1$, let $g_1={\beta }_{11}{\displaystyle \frac{x_1^2}{x^2}}+{\beta }_{12}{\displaystyle \frac{x_1{x}_2}{x^2}}+{\beta }_{22}{\displaystyle \frac{x_2^2}{4x^2}}$. The partial derivative is:

$${\displaystyle \frac{\partial f_1}{\partial x_1}}={\alpha }_1{g}_1+{\alpha }_{1}x\left({\beta }_{11}{\displaystyle \frac{2x_1{x}^2-x_1^2·2x}{x^4}}+{\beta }_{12}{\displaystyle \frac{x_2{x}^2-x_1{x}_2·2x}{x^4}}\right)-{\displaystyle \frac{{\alpha }_1}{K}}(x+x_1)-m_{1}y{\displaystyle \frac{(a+x)x-x_{1}x}{{(a+x)}^2}}-d_1$$

Since $x\ge {\delta }_0$, $x_i,y\le M$, and $a+x\ge a$, each term is bounded. For example, ${\displaystyle \frac{2x_1(x-x_1)}{x^3}}\le {\displaystyle \frac{2M·2M}{{\delta }_0^3}}={\displaystyle \frac{4M^2}{{\delta }_0^3}}$. Similarly, ${\displaystyle \frac{\partial f_1}{\partial x_2}},{\displaystyle \frac{\partial f_1}{\partial x_3}},{\displaystyle \frac{\partial f_1}{\partial y}}$ involve rational functions bounded on $D_{loc}$. Since all the terms are in the same form, we have demonstrated continuity in the case of one term; the rest of the terms follow analogously. For $f_4$

$${\displaystyle \frac{\partial f_4}{\partial y}}={\displaystyle \frac{m_1{x}_1+m_2{x}_2}{a+x}}-s-{\displaystyle \frac{m_3{x}_3}{a+x}}\le {\displaystyle \frac{m_{1}M+m_{2}M}{a}}+s+{\displaystyle \frac{m_{3}M}{a}},$$

which is finite. The control term in $f_3$, $-u{x}_3$, has ${\displaystyle \frac{\partial }{\partial x_3}}(-u{x}_3)=-u\le u_{max}$. Thus, ${\displaystyle \frac{\partial f}{\partial \mathbf{x}}}$ is bounded, implying Lipschitz continuity with constant $L>0$. By the Picard-Lindelöf theorem, there exists $\delta >0$ such that a unique solution $\mathbf{x}\left(t\right)\in C^1([0,\delta ];D_{loc})\subset C^1([0,\delta ];D)$ exists. Hence, the theorem is proved.

For our optimal control problem to be mathematically well-posed and biologically meaningful, we show the global existence of solutions. □

Theorem 2.

Under the assumptions of Theorem 1, with $u\left(t\right)\in [0,u_{max}]$ and the parameter condition $max(m_1,m_2)x_{max}<sa$ where $x_{max}=max\left(x\left(0\right),{\displaystyle \frac{K[C{max}_i\left({\alpha }_i\right)-{min}_i\left(d_i\right)]}{{min}_i\left({\alpha }_i\right)}}\right)$ with $C=4{max}_{i,j}\left|{\beta }_{ij}\right|$, the solution $\mathbf{x}\left(t\right)$ exists for all $t\ge 0$ and remains bounded in D with $x\left(t\right)\le min(K,x_{max})$.

Proof. 

We establish global existence through three main steps as follows: (i) positive invariance and non-negativity, (ii) boundedness of the total prey population, and (iii) boundedness of the predator population.

(i). Positive Invariance and Non-negativity

The non-negative orthant ${\mathbb{R}}_{+}^4=\{(x_1,x_2,x_3,y)\in {\mathbb{R}}^4:x_i\ge 0,y\ge 0\}$ is positively invariant under the flow of system (1)–(4). Moreover, the interior ${\mathbb{R}}_{++}^4=\{(x_1,x_2,x_3,y)\in {\mathbb{R}}^4:x_i>0,y>0\}$ is positively invariant.

We prove invariance by showing that the vector field points inward (or tangentially) on each face of the boundary of ${\mathbb{R}}_{+}^4$. We start with when $x_1=0$, we have $x=x_2+x_3$. The first equation becomes:

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{d{x}_1}{dt}}\right|}_{x_1=0}& ={\alpha }_{1}x\left[0+0+{\displaystyle \frac{{\beta }_{22}}{4}}{\left({\displaystyle \frac{x_2}{x}}\right)}^2\right]-0-0-0\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{{\alpha }_1{\beta }_{22}{x}_2^2}{4x}}\hfill \end{array}$$

(6)

Since $x_2\ge 0$ and $x=x_2+x_3\ge 0$, we have ${\left.{\displaystyle \frac{d{x}_1}{dt}}\right|}_{x_1=0}\ge 0$. Equality holds if and only if $x_2=0$.

When $x_2=0$, we have $x=x_1+x_3$. The second equation becomes

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{d{x}_2}{dt}}\right|}_{x_2=0}& =2{\alpha }_{2}x\left[{\beta }_{13}{\displaystyle \frac{x_1{x}_3}{x^2}}+0+0+0\right]-0-0-0\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{2{\alpha }_2{\beta }_{13}{x}_1{x}_3}{x}}\hfill \end{array}$$

(8)

Since $x_1,x_3\ge 0$ and $x>0$ (when not all variables are zero), we have ${\left.{\displaystyle \frac{d{x}_2}{dt}}\right|}_{x_2=0}\ge 0$.

When $x_3=0$, we have $x=x_1+x_2$. The third equation becomes

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{d{x}_3}{dt}}\right|}_{x_3=0}& ={\alpha }_{3}x\left[0+0+{\displaystyle \frac{1}{4}}{\beta }_{22}{\left({\displaystyle \frac{x_2}{x}}\right)}^2\right]-0-0-0-0\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{{\alpha }_3{\beta }_{22}{x}_2^2}{4x}}\ge 0\hfill \end{array}$$

(10)

Since $x_2\ge 0$ and $x=x_1+x_2\ge 0$, we have ${\left.{\displaystyle \frac{d{x}_3}{dt}}\right|}_{x_3=0}\ge 0$. Equality holds if and only if $x_2=0$.

When $y=0$, the fourth equation becomes

$$\begin{array}{cc}\hfill {\left.{\displaystyle \frac{dy}{dt}}\right|}_{y=0}& =0·\left[{\displaystyle \frac{m_1{x}_1+m_2{x}_2}{a+x}}-s\right]-0=0\hfill \end{array}$$

(11)

For edges where two coordinates are zero, the analysis follows similarly. On each face of the boundary, the outward normal derivative is non-negative, which means the vector field either points into the domain or is tangent to the boundary. This ensures that ${\mathbb{R}}_{+}^4$ is positively invariant.

For the interior ${\mathbb{R}}_{++}^4$, we need to show that if $(x_1\left(0\right),x_2\left(0\right),x_3\left(0\right),y\left(0\right))\in {\mathbb{R}}_{++}^4$, then the solution remains in ${\mathbb{R}}_{++}^4$ for all $t>0$.

Suppose, for contradiction, that there exists a first time $T>0$ such that one of the coordinates reaches zero. Without loss of generality, assume $x_1\left(T\right)=0$ and $x_1\left(t\right)>0$ for $t\in [0,T)$. Since $x_1\left(t\right)>0$ for $t\in [0,T)$ and $x_1\left(T\right)=0$, we must have $x_1^{\prime }\left(T\right)\le 0$. However, from Step 1, we showed that $x_1^{\prime }\left(T\right)\ge 0$ when $x_1\left(T\right)=0$. The only way both conditions can be satisfied is if $x_1^{\prime }\left(T\right)=0$, which occurs when $x_2\left(T\right)=0$. By similar reasoning applied to $x_2$, we would need $x_1\left(T\right)=0$ and $x_3\left(T\right)=0$. Continuing this argument leads to all coordinates being zero simultaneously, which contradicts the continuity of solutions unless the initial condition was the origin.

For the y coordinate, if $y\left(t\right)>0$ for $t\in [0,T)$ and $y\left(T\right)=0$, then $y^{\prime }\left(T\right)\le 0$. But from Step 4, $y^{\prime }\left(T\right)=0$ when $y\left(T\right)=0$, which is consistent. Therefore, ${\mathbb{R}}_{++}^4$ is positively invariant under the assumption that not all prey species go extinct simultaneously. □

(ii). Boundedness of Total Prey Population

Let $x=x_1+x_2+x_3$ denote the total prey population. We compute $\dot{x}={\dot{x}}_1+{\dot{x}}_2+{\dot{x}}_3$ and analyze the boundedness by examining each component of the vector field $f_i$ i.e., (1)–(3).

The growth terms are bounded by

$$\begin{array}{cc}\hfill {\alpha }_{1}x\sum {\beta }_{ij}{\displaystyle \frac{x_i{x}_j}{x^2}}& \le {\alpha }_1{\beta }_{max}x,\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill 2{\alpha }_{2}x\sum {\beta }_{ij}{\displaystyle \frac{x_i{x}_j}{x^2}}& \le 2{\alpha }_2{\beta }_{max}x,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\alpha }_{3}x\sum {\beta }_{ij}{\displaystyle \frac{x_i{x}_j}{x^2}}& \le {\alpha }_3{\beta }_{max}x,\hfill \end{array}$$

(14)

where ${\beta }_{max}={max}_{i,j}\left|{\beta }_{ij}\right|$. The total growth contribution is at most $C{max}_i\left({\alpha }_i\right)x$ with $C=4{\beta }_{max}$. The carrying capacity terms satisfy

$$-{\displaystyle \frac{1}{K}}({\alpha }_1{x}_{1}x+{\alpha }_2{x}_{2}x+{\alpha }_3{x}_{3}x)=-{\displaystyle \frac{x^2}{K}}\sum {\alpha }_i{\displaystyle \frac{x_i}{x}}\le -{\displaystyle \frac{{min}_i\left({\alpha }_i\right)}{K}}{x}^2.$$

The predation, death, and control terms contribute

$$-\sum _{i=1}^3{\displaystyle \frac{x_i{m}_{i}xy}{a+x}}-\sum _{i=1}^3{d}_i{x}_i-u{x}_3\le -\underset{i}{min}\left(d_i\right)x,$$

since $u\ge 0$. Combining these estimates yields a differential inequality

$$\dot{x}\le C\underset{i}{max}\left({\alpha }_i\right)x-{\displaystyle \frac{{min}_i\left({\alpha }_i\right)}{K}}{x}^2-\underset{i}{min}\left(d_i\right)x.$$

(15)

The right-hand side is a quadratic in x with negative leading coefficient $-{\displaystyle \frac{{min}_i\left({\alpha }_i\right)}{K}}$.

Setting the right-hand side equal to zero gives the equilibrium point

$$x^{*}={\displaystyle \frac{K[C{max}_i\left({\alpha }_i\right)-{min}_i\left(d_i\right)]}{{min}_i\left({\alpha }_i\right)}}.$$

To establish boundedness, we prove that $x\left(t\right)\le max(x\left(0\right),x^{*})$ for all $t\ge 0$. We proceed by considering two cases based on the initial conditions.

Define

$$g\left(x\right)=C\underset{i}{max}\left({\alpha }_i\right)x-{\displaystyle \frac{{min}_i\left({\alpha }_i\right)}{K}}{x}^2-\underset{i}{min}\left(d_i\right)x,$$

where $g\left(x\right)$ is a downward-opening parabola with roots at $x=0$ and the other root is $x^{*}={\displaystyle \frac{K[C{max}_i\left({\alpha }_i\right)-{min}_i\left(d_i\right)]}{{min}_i\left({\alpha }_i\right)}}$.

We observe that $g\left(x\right)>0$ for $x\in (0,x^{*})$ and $g\left(x\right)<0$ for $x>x^{*}$. This gives two cases for the initial condition.

Firstly, if $x\left(0\right)\le x^{*}$, then we show that $x\left(t\right)\le x^{*}$ for all $t\ge 0$. Suppose that there exists a first time $T>0$ such that $x\left(T\right)>x^{*}$. By continuity of solutions, there must exist $T_0\in (0,T)$ such that $x\left(T_0\right)=x^{*}$ and $x\left(t\right)<x^{*}$ for $t\in [0,T_0)$. Since $x\left(t\right)$ crosses the level $x^{*}$ from below at $t=T_0$, we have $\dot{x}\left(T_0\right)\ge 0$. However, the differential inequality gives $\dot{x}\left(T_0\right)\le g\left(x^{*}\right)=0$. For any time immediately after $T_0$ where $x\left(t\right)>x^{*}$ we have $g\left(x\right)<0$ which implies $\dot{x}\left(t\right)<0$. Therefore, $x\left(t\right)$ must be decreasing right after $T_0$, contradicting the existence of $T>T_0$ with $x\left(T\right)>x^{*}$.

Secondly, if $x\left(0\right)>x^{*}$ then we show that $x\left(t\right)\le x\left(0\right)$ for all $t\ge 0$. For any $t>0$ such that $x\left(t\right)>x^{*}$, the differential inequality yields $\dot{x}\left(t\right)\le g\left(x\left(t\right)\right)<0$ since $g\left(x\right)<0$ for $x>x^{*}$. This implies that $x\left(t\right)$ is strictly decreasing whenever $x\left(t\right)>x^{*}$, and therefore $x\left(t\right)$ cannot exceed its initial value $x\left(0\right)$.

Combining both cases establishes that

$$x\left(t\right)\le x_{max}=max(x\left(0\right),x^{*})\mathrm{for}\mathrm{all}t\ge 0.$$

This establishes the desired boundedness of the total prey population.

(iii). Boundedness of Predator population Similarly for the predator we have, since $x\le x_{max}$

$$\dot{y}=y\left({\displaystyle \frac{m_1{x}_1+m_2{x}_2}{a+x}}-s\right)-{\displaystyle \frac{m_3{x}_{3}y}{a+x}}\le y\left({\displaystyle \frac{max(m_1,m_2)x}{a}}-s\right).$$

By assumption, $max(m_1,m_2)x_{max}<sa$, so

$$\dot{y}\le y\left({\displaystyle \frac{max(m_1,m_2)x_{max}}{a}}-s\right)=-ky,$$

where $k=s-{\displaystyle \frac{max(m_1,m_2)x_{max}}{a}}>0$. Thus, $y\left(t\right)\le y\left(0\right)e^{-kt}+\mathrm{bounded}\mathrm{terms}$, ensuring y is bounded.

Since $\mathbf{x}\left(t\right)$ is non-negative, $x\le x_{max}$, $y\le y_{max}$, and $x_1+x_2+x_3\ge \delta$, solutions remain in a compact subset of D. By Lipschitz continuity of f, solutions cannot blow up in finite time, so they exist for all $t\ge 0$. Theorems 1 and 2 ensure the problem is well-posed, enabling further analysis of optimal control strategies.

3.2. Dynamical Behavior of the Uncontrolled System

We provide a brief description of the model’s equilibrium points. For optimal control applications, we have chosen parameter values (following [13]) that enable the system to operate in two distinct regimes as follows: one characterized by sustained oscillations and another where oscillations are damped and eventually cease due to the extinction of the predator. We plot in Figure 2a the extrema of the trajectories of the mutant prey ($x_3$) and the predator (y) with respect to the mutant prey genotype mortality rate parameter ($d_3$). The sustained oscillation regime can be clearly seen after a certain value of $d_3$ (the death rate of $x_3$), i.e, clearly visible from the bifurcation type plot Figure 2a. This clearly shows that if the $x_3$ values are low, the predator survives, as seen for higher values of $d_3$. The asymptotic values of the extrema are shown in red. For further verification of the phenomenon, we did the same analysis for the joint parameter values of $m_3$ and $d_3$.

The equilibrium points of these type of genetic systems without predators have been well studied in [17] and genetic systems with predator influences (eco-evolutionary dynamics) have been studied in [15,18,19], focusing on the stability of equilibria, persistence, and extinctions. We present the equilibrium points here for completeness and a self-contained understanding of the system’s dynamical behavior.

The trivial equilibrium point is (0,0,0,0). Next, we describe the predator-free points with $y=0$. If all the $d_i$ ’s are not equal, then we have the two boundary equilibrium points as $(1-{\displaystyle \frac{d_1}{\alpha }},0,0,0)$ and $(0,0,1-{\displaystyle \frac{d_3}{\alpha }},0)$. There exist another equilibrium point which is also a mutant free point i.e., $x_2=x_3=0$ as given by $(x_1^{*},0,0,y^{*})$ where $x_1^{*}={\displaystyle \frac{as}{m_1-s}}$ and $y^{*}={\displaystyle \frac{(a\alpha (m_1-as-s)+a{d}_1(s-m_1))}{{(m_1-s)}^2}})$.

Further, there is one more point in the predator-free regime, i.e., $y=0$, which is the interior Hardy–Weinberg equilibrium point given by

$$(x_1^{*},x_2^{*},x_3^{*},0):=E^{*}=E^{*}\left(c\right)=\left({\displaystyle \frac{c^{2}K}{{(1+c)}^2}},{\displaystyle \frac{2cK}{{(1+c)}^2}},{\displaystyle \frac{K}{{(1+c)}^2}},0\right);\forall c\in (0,\infty );K=1-{\displaystyle \frac{d}{\alpha }}$$

An analytically tractable expression for this equilibrium can be obtained in the special case where ${\alpha }_i=\alpha$ and $d_1=d_2=d_3=d$, as given in [17]. In the above setting of ${\alpha }_i=\alpha \mathrm{and}{d}_1=d_2=d_3=d$, by Theorem 4.2 in [17], we have

$$\underset{t\to \infty }{lim}\left(x_1\left(t\right),x_2\left(t\right),x_3\left(t\right)\right)=E^{*}\left(c\right)\mathrm{where}c={\displaystyle \frac{x_1\left(0\right)+\frac{1}{2}{x}_2\left(0\right)}{x_3\left(0\right)+\frac{1}{2}{x}_2\left(0\right)}}$$

We show the equilibrium of the form $(x_1^{*},x_2^{*},x_3^{*},0)$ in the nullcline analysis in the Figure 3a for the parameter range selected for control. In Figure 3b, we show the asymptotic behavior of the dynamical system by plotting phase trajectories.

3.3. Optimal Control Formulation

Having established the existence and uniqueness of solutions to the controlled population dynamics system (Theorems 1 and 2), we formulate and analyze an optimal control problem to suppress the mutant population ($x_3$) while maintaining ecological stability. We apply Pontryagin’s Maximum Principle (PMP) to derive the necessary conditions for optimal control, providing a framework for numerical implementation due to the system’s high nonlinearity. We have followed Ch. 4–5 of [11] and Ch. 2, Section 5, 12, and 15 in [33] for the methods used in the proofs. The optimal control problem is thus given as follows.

Objective Functional: We define the objective functional as

$$J\left(u\right)={\int }_0^T\left[q{x}_3^2+c{u}^2\right]dt+p{\left[x_3\left(T\right)\right]}^2,$$

(16)

where $q,c,p>0$ are weights that penalize the mutant population, control effort, and the terminal cost, respectively.

Hamiltonian Formulation: The Hamiltonian is given as

$$\mathcal{H}(\mathbf{x},u,\mathit{\lambda })=q{x}_3^2+c{u}^2+\sum _{i=1}^4{\lambda }_i{f}_i(\mathbf{x},u),$$

(17)

where $\mathit{\lambda }={[{\lambda }_1,{\lambda }_2,{\lambda }_3,{\lambda }_4]}^T\in {\mathbb{R}}^4$ is the costate vector. The PMP states that if $u^{*}\left(t\right)$ is optimal with the corresponding state ${\mathbf{x}}^{*}\left(t\right)$, there exists costate vector $\mathit{\lambda }\left(t\right)$ that satisfies the following necessary conditions for optimality for almost all $t\in [0,T]$.

The state dynamics are given by

$${\dot{\mathbf{x}}}^{*}\left(t\right)={\displaystyle \frac{\partial \mathcal{H}}{\partial \mathit{\lambda }}}=f({\mathbf{x}}^{*}\left(t\right),u^{*}\left(t\right)),{\mathbf{x}}^{*}\left(0\right)={\mathbf{x}}_0.$$

The costate dynamics thus become

$${\dot{\lambda }}_i\left(t\right)=-{\displaystyle \frac{\partial \mathcal{H}}{\partial x_i}}({\mathbf{x}}^{*}\left(t\right),u^{*}\left(t\right),\mathit{\lambda }\left(t\right)),i=1,2,3,4,$$

with terminal conditions from the terminal cost $\varphi \left(\mathbf{x}\left(T\right)\right)=p{\left[x_3\left(T\right)\right]}^2$

$${\lambda }_i\left(T\right)={\displaystyle \frac{\partial \varphi }{\partial x_i}}\left(\mathbf{x}\left(T\right)\right)=\left\{\begin{array}{cc}2p{x}_3\left(T\right)\hfill & \mathrm{if}i=3,\hfill \\ 0\hfill & \mathrm{if}i=1,2,4.\hfill \end{array}\right.$$

Finally, using (17), we get the costate equations as

$$\begin{array}{cc}\hfill {\dot{\lambda }}_1& =-\sum _{j=1}^4{\lambda }_j{\displaystyle \frac{\partial f_j}{\partial x_1}}\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\dot{\lambda }}_2& =-\sum _{j=1}^4{\lambda }_j{\displaystyle \frac{\partial f_j}{\partial x_2}}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\dot{\lambda }}_3& =-2q{x}_3-\sum _{j=1}^4{\lambda }_j{\displaystyle \frac{\partial f_j}{\partial x_3}}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\dot{\lambda }}_4& =-\sum _{j=1}^4{\lambda }_j{\displaystyle \frac{\partial f_j}{\partial y}}\hfill \end{array}$$

(21)

where $f_j$ are from (1)–(4).

Minimization Condition: The optimal control is characterized by the minimization condition given by

$$u^{*}\left(t\right)=arg\underset{u\in [0,u_{max}]}{min}\mathcal{H}({\mathbf{x}}^{*}\left(t\right),u,\mathit{\lambda }\left(t\right))$$

(22)

To determine the optimal control, we minimize the Hamiltonian $\mathcal{H}$ with respect to u, keeping all other variables fixed,

$$\mathcal{H}=q{x}_3^2+c{u}^2+{\lambda }_1{f}_1+{\lambda }_2{f}_2+{\lambda }_3(f_3-u{x}_3)+{\lambda }_4{f}_4.$$

Taking the derivative with respect to u and setting it to zero gives the unconstrained minimizer

$${\displaystyle \frac{\partial \mathcal{H}}{\partial u}}=2cu-{\lambda }_3{x}_3=0\Rightarrow u={\displaystyle \frac{{\lambda }_3{x}_3}{2c}}.$$

Projecting this onto the admissible control set $u\in [0,u_{max}]$ yields the optimal control as follows:

$$u^{*}\left(t\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{\displaystyle \frac{{\lambda }_3\left(t\right)x_3^{*}\left(t\right)}{2c}}<0,\hfill \\ {\displaystyle \frac{{\lambda }_3\left(t\right)x_3^{*}\left(t\right)}{2c}}\hfill & \mathrm{if}0\le {\displaystyle \frac{{\lambda }_3\left(t\right)x_3^{*}\left(t\right)}{2c}}\le u_{max},\hfill \\ u_{max}\hfill & \mathrm{if}{\displaystyle \frac{{\lambda }_3\left(t\right)x_3^{*}\left(t\right)}{2c}}>u_{max}.\hfill \end{array}\right.$$

(23)

When the control lies strictly within bounds and the switching function

$${\displaystyle \frac{\partial \mathcal{H}}{\partial u}}=2cu-{\lambda }_3{x}_3,$$

vanishes identically over a nontrivial time interval, and the system enters a singular arc. In such cases, higher-order necessary conditions—such as the vanishing of time derivatives of the switching function—must be evaluated. Due to the nonlinearity of the system, such situations are generally resolved numerically.

3.4. Numerical Results and Performance Evaluation

We have solved the optimal control problem numerically to demonstrate ecological rescue of the predator population (y) by suppressing the mutant population ($x_3$). By solving the two-point boundary value problem (TPBVP), we compared the controlled and uncontrolled system dynamics to evaluate the effectiveness of the control strategy. We have used biologically valid parameters following [13] ${\alpha }_i=1.2$, ${\beta }_{ij}=1$, $K=1$, $m_1=m_2=2.5$, $m_3=2.25$, $a=0.37$. Initial conditions are $\mathbf{x}\left(0\right)={[0.02,0.002,0.00001,0.1]}^T$, reflecting a small initial mutant population. We make two cases as follows: one for $s=0.81$ and $d_1=d_2=d_3=0.0591$ and the second one as $s=0.91$ and $d_1=d_2=d_3=0.0591$. We have plotted the dynamics in Figure 4 and Figure 5, respectively.

The time horizon $[0,1000]$ is discretized into $N=1000$ intervals ($\Delta t=1$). The control $u\left(t\right)\in [0,1]$ allows a full range of suppression efforts. The performance index is as follows:

$$J\left(u\right)={\int }_0^T\left[q{(x_3^{\mathrm{target}}-x_3)}^2+r{x}_3^2+c{u}^2\right]dt+p{(x_3^{\mathrm{target}}-x_3\left(T\right))}^2,$$

with $q=3.0$, $r=3.0$, $c=0.1$, $p=10$, and $x_3^{\mathrm{target}}=0.001$. The terms $q{(x_3^{\mathrm{target}}-x_3)}^2+r{x}_3^2$ minimize $x_3$, $c{u}^2$ limits control effort, and the terminal cost reinforces the suppression of $x_3\left(T\right)$. The simulation uses CasADi’s Opti stack, with the control sequence $U={[u\left(t_0\right),u\left(t_1\right),\dots ,u\left(t_{N-1}\right)]}^T\in {\mathbb{R}}^N$ as decision variables. State trajectories are computed via

$$\mathbf{x}\left(t_{k+1}\right)=F(\mathbf{x}\left(t_k\right),u\left(t_k\right)),$$

where F is the CVODES integrator solving (1)–(4) over $[t_k,t_{k+1}]$. The cost is approximated as

$$J\approx \sum _{k=0}^{N-1}L(\mathbf{x}\left(t_k\right),u\left(t_k\right))\Delta t+p{(x_3^{\mathrm{target}}-x_3\left(t_N\right))}^2,$$

where $L=q{(x_3^{\mathrm{target}}-x_3)}^2+r{x}_3^2+c{u}^2+ϵ(x_1^2+x_2^2)$. Control constraints $0\le u\left(t_k\right)\le 1$ are enforced, and the NLP is solved using IPOPT (tolerance ${10}^{-2}$, maximum iterations 500, MUMPS linear solver). The initial control guess is $u=0.1$.

The numerical simulations comprehensively illustrate the predator–prey system’s dynamics with and without optimal control, highlighting the ecological rescue of the predator y under two parameter sets as follows: Figure 4 for set $s=0.81$, $d=0.0591$ and Figure 5 for the set $s=0.91$, $d=0.0591$. In the uncontrolled scenario in both the cases as shown in Figure 4 and Figure 5a,b,f, the mutant prey population $x_3$ rapidly proliferates due to its relatively lower capture rate, and due to its harmful nature, it significantly increases predation pressure on y. This unchecked growth drives y toward extinction, as evidenced by the phase plot Figure 4f and Figure 5f, where trajectories spiral inward to the axis $y\to 0$, signaling a collapse of predator–prey interactions. To counteract this decline, an optimal control strategy $u\left(t\right)$ is applied, effectively suppressing $x_3$ to a predefined target threshold while minimizing a cost function as given in the above section. Under control, the dynamics are plotted in Figure 4 and Figure 5c, e.g., $x_3$ is regulated near the target, reducing the predation pressure and enabling y to stabilize above zero in both cases. In Figure 5 for the parameter set ($s=0.91$, $d=0.0591$) we show how the control signal is able to maintain sustained oscillations in $x_1$ and y (Figure 5g), with the phase plot reflecting a healthy limit cycle. These simulations, computed over $t\in [0,1000]$ with $N=1000$ intervals, underscore the control’s efficacy in preventing extinction across varying conditions while highlighting the pivotal role of s and d in determining the strength and sustainability of predator–prey interactions.

3.5. Control Efficacy and Population Dynamics

The optimal control strategy was applied to suppress the mutant prey population ($x_3$, aa genotype) below a target threshold of 0.001 while maintaining the predator population (y) near 0.21, as governed by the dynamics in (1)–(4), with the parameters defined in the previous section. The controlled system achieved a final $x_3=0.000982$, meeting the suppression target, compared to an uncontrolled final $x_3=0.229092$. The predator population stabilized at $y=0.163517$ in the controlled case, approaching the target of 0.21, whereas the uncontrolled system resulted in predator extinction ($y=0.0$).

The basic reproduction number ($R_0^{x_3}$) is shown in Figure 6A for the mutant prey was significantly reduced under control. In the uncontrolled system, the mean $R_0^{x_3}=57.1429$, reflecting high intrinsic growth potential absent ecological constraints [50]. With control, the mean $R_0^{x_3}=12.2989$, reflecting a reduction of 44.8439, attributed to the increased loss rate from the control input $u\left(t\right)$. Control efficacy was assessed using a composite metric, weighting the proximity of $x_3$ to its target (weight 0.8), y to its target (weight 0.1), and a penalty for control effort (weight 0.1) as shown in Figure 6B. The mean control efficacy was 81.07%, with a maximum of 89.87%, indicating effective suppression of $x_3$. The mean efficacy was below 100% due to oscillations, where $x_3\left(t\right)>0.001$ and $y\left(t\right)$ deviated from 0.21 (initially $y\left(0\right)=0.1$). The effective reproduction number ($R_e^{x_3}$) is shown in Figure 6C, which accounts for competition, predation, and control, was further suppressed [48,49]. The controlled system yielded a mean $R_e^{x_3}=0.1115$, below the threshold of 1, confirming ecological suppression of $x_3$. In contrast, the uncontrolled mean $R_e^{x_3}=0.6599$, with a difference of 0.5484, highlights the control’s role in limiting mutant proliferation.

These results, summarized in

Table 2. Summary of control performance metrics for mutant prey suppression.

Metric	Uncontrolled	Controlled	Target
Final $x_3$	0.229092	0.000982	0.001000
Final y	0.000000	0.163517	0.210000
Mean Efficacy (%)	–	81.07	–
Max Efficacy (%)	–	89.87	–
Mean $R_0^{x_3}$	57.1429	12.2989	–
Mean $R_e^{x_3}$	0.6599	0.1115	–

, demonstrate the efficacy of the optimal control strategy in suppressing the mutant prey population while supporting a stable predator population. The low $R_e^{x_3}$ in the controlled system accounts for the successful reduction of $x_3$, despite a high $R_0^{x_3}$, underscoring the combined effects of predation, competition, and targeted control.

4. Discussion

This study demonstrates the ecological rescue of a predator population (y) threatened by a mutant population ($x_3$) in a four-species nonlinear system. Numerical simulations reveal that uncontrolled dynamics lead to $x_3$ dominance, predator extinction, and oscillation collapse, while the optimal control, derived via Pontryagin’s Maximum Principle (PMP), suppresses $x_3$, sustains y, and restores oscillatory dynamics. These findings underscore the importance of targeted interventions to maintain predator–prey cycles, critical for ecological stability. The proofs of local existence and uniqueness (Theorem 1) and global boundedness (Theorem 2) ensure the model’s mathematical robustness, extending traditional ecological models with genetic and control elements. However, constant parameters and fixed control bounds simplify real-world variability and resource constraints. Numerical reliance due to nonlinearity poses computational challenges. Future work could incorporate stochasticity, spatial dynamics, or multi-input controls and validate the model with field data from invasive species scenarios. This framework informs conservation strategies, offering a mathematically rigorous approach to managing ecological disruptions and preserving biodiversity.

5. Conclusions

This work presents a robust framework for ecological rescue by controlling a mutant population in a four-species nonlinear system. Through a nonlinear ODE model, we prove the local and global existence of solutions, ensuring biological realism. Applying PMP, we derive an optimal control that minimizes the mutant population, rescues the predator, and restores ecological oscillations, as validated by numerical simulations. The contrast between uncontrolled (predator extinction, oscillation collapse) and controlled (rescue, oscillation restoration) dynamics highlights the control’s efficacy. This study bridges the fields of applied mathematics and ecology, providing a versatile tool for managing mutant populations. Its applications extend to conservation, invasive species control, and ecosystem restoration, with potential in microbial or disease dynamics. Future research could explore time-varying parameters, spatial models, or empirical validation to enhance practical impact. This framework advances our ability to design precise interventions for sustainable ecological management.