The Hamilton–Jacobi–Bellman Equation for Differential Games with Composite Distribution of Random Time Horizon

Abstract

A differential game with random duration is considered. The terminal time of the game is a random variable settled using a composite distribution function. Such a scenario occurs when the operating mode of the system changes over time at the appropriate switching points. On each interval between switchings, the distribution of the terminal time is characterized by its own distribution function. A method for solving such games using dynamic programming is proposed. An example of a non-renewable resource extraction model is given, where a solution of the problem of maximizing the total payoff in closed-loop strategies is found. An analytical view of the optimal control of each player and the optimal trajectory depending on the parameters of the described model is obtained.

1. Introduction

Differential games are widely used to model conflict-controlled processes that develop continuously in time. If the end time of the game is known, then the game is considered to be on a finite time interval. It is also common to consider games with an infinite horizon. However, when trying to describe real life processes, one often encounters uncertainty, in the sense that the terminal time of the game is not known in advance, but is the realization of some random variable. Such games are called games with random duration. For the first time, optimal control problems with uncertain lifetime were considered by Yaari in [1]. In this work, the use of such models was explained by the fact that the consumer, making plans for the future, one way or another, must take into account that he does not know how long he will live. Similar issues have been considered by Boukas in [2]. Later, this idea was widely applied in problems of dynamic games. The study of cooperative and non-cooperative differential games with random duration is presented by Shevkoplyas and Petrosyan in [3,4]. In work [5], for example, the randomness of the terminal time was explained by the life circle of technical equipments.

In some cases, the probability density function of the terminal time can change depending on different conditions and the standard distribution cannot fully simulate the random variable responsible for the moment of the end of the game. Such a scenario occurs when the operating mode of the system changes over time at the appropriate switching points and is characterized by its own distribution at each individual interval between switching. It can be used in financial models due to a possible financial credit crisis, as well as in modeling possible defaults and credit risks. Also in environmental models due to possible environmental disasters and climate change or in technical models due to equipment failures or different modes of technical equipment. In such problems, a composite distribution function for terminal time is used. By Zaremba et al. in [6], it is assumed that the duration of the game is a random variable corresponding to the discontinuous cumulative distribution function. Gromov and Gromova [7,8] considered games with random horizon and composite distribution function for terminal time as hybrid differential games, since payoffs of players in such games take the form of the sums of integrals with different, but adjoint time intervals. In [9], the differential game with composite distribution function for terminal time with two switching moments is investigated. Kamien and Schwartz [10] also used a composite distribution function in the model of the firm’s timing of a supramarginal technically modest innovation in a setting of rivalrous competition. Such function is used there to reflect the possibility that if one firm is first to introduce the product, rivals may alter their development rates. Other variants of models with switches at a random time moment can be found in [11]. In all these works, the solutions were found in the class of open-loop strategies using the maximum Pontryagin principle. However, since we are considering a process with many participants the players can deviate from some agreed course of action. In addition, only using the feedback strategies players can have a chance to react if they observe some deviations.

The purpose of this work is studying of games with a composite distribution function for terminal time in the class of feedback strategies using the dynamic programming methods [12]. The methods are illustrated on a class of optimal allocation models. As was mentioned in [13,14] such models found a lot of applications in economics in various forms: exhaustible resource extraction [15], cake-eating [16], and life-cycle saving [1]. The modern resource management has to take account of the possibility of regime shifts. In [17], for instance, a renewable resource model with the risk of regime shifts is considered. In such models, the state variable is the level of resources remaining. In addition, it is usually assumed that the utility functions of players are strictly concave and increasing. The relevance of the chosen model is confirmed, for example, by successful applying the resource extraction model to the coal mining problem in the form of a three-player differential cooperative game, where the players are the real companies with real life parameters in [18].

The paper is organized as follows. Section 2 gives the game formulation and the basic assumptions of the model. The following Section 3 presents a method for solving such games. The system of Hamilton–Jacobi–Bellman equations for the problem of maximizing the total payoff is given here. An illustrative example of a non-renewable resource extraction model is investigated in Section 4. The case of time-dependent switch is studied in Section 4.1. It is assumed that the switching moment of the composite distribution function is determined in advance. Section 4.2 demonstrates the state-dependent case, where switching time is not fixed, switching occurs when the system reaches a certain state.

2. Problem Statement

Consider differential n-player game ${\mathsf{\Gamma }}^T(x_0,{\tau }_0)$ presented by Gromov and Gromova in [7]. The game starts from initial state $x_0$ at the time ${\tau }_0$ and the terminal time T is the random variable, distributed on the time interval $[{\tau }_0;\infty )$ according to known composite distribution function $F\left(t\right)$ with $N-1$ switches at fixed moments of time $\tau :t_0={\tau }_0<{\tau }_1<\dots <{\tau }_{N-1}<{\tau }_N=\infty$. Composite distribution function $F\left(t\right)$ is defined as follows:

$$F\left(t\right)=\left\{\begin{array}{cc}0,\hfill & t<{\tau }_0,\hfill \\ {\alpha }_j\left({\tau }_j\right)F_{j+1}\left(t\right)+{\beta }_j\left({\tau }_j\right),\hfill & t\in [{\tau }_j,{\tau }_{j+1}),0\le j\le N-1.\hfill \end{array}\right.$$

(1)

Here, $\{F_j\left(t\right),j=1,\dots ,N\}$ is the set of distribution functions characterizing different modes of operation. $F_j\left(t\right)$ are assumed to be absolutely continuous nondecreasing functions such that each distribution function converges to 1 asymptotically, i.e., $F_j\left(t\right)<1$$\forall t<\infty .$

To guarantee that composite distribution function $F\left(t\right)$ is a continuous function suppose that $F_1\left({\tau }_0\right)=0$, ${\alpha }_j\left({\tau }_j\right)=\frac{F\left({\tau }_j^{-}\right)-1}{F_{j+1}\left({\tau }_j\right)-1}$, ${\beta }_j\left({\tau }_j\right)=1-\frac{F\left({\tau }_j^{-}\right)-1}{F_{j+1}\left({\tau }_j\right)-1}.$ Note that $F\left({\tau }_j^{-}\right)$ is defined as the left limit $F\left(t\right)$ at $t={\tau }_j^{-}$, i.e., $F\left({\tau }_j^{-}\right)=\underset{t\to ({\tau }_j-0)}{lim}F\left(t\right)$.

Further, we assume an existence of composite probability density function, that is defined as the derivative of distribution function, and has the following form:

$$f\left(t\right)={\alpha }_j\left({\tau }_j\right)f_{j+1}\left(t\right),t\in [{\tau }_j,{\tau }_j+1),1\le j\le N-1.$$

(2)

For simplicity, we denote ${\overline{F}}_{j+1}\left(t\right)={\alpha }_j\left({\tau }_j\right)F_{j+1}\left(t\right)+{\beta }_j\left({\tau }_j\right)$ and ${\overline{f}}_{j+1}\left(t\right)={\alpha }_j\left({\tau }_j\right)f_{j+1}\left(t\right).$

Let $h_i\left(u\left(t\right)\right)\ge 0$ be an instantaneous payoff of the player i in the game ${\mathsf{\Gamma }}^T(x_0,{\tau }_0)$ at time t, $u\left(t\right)=(u_1\left(t\right),\dots ,u_n\left(t\right))$, $u_i\in U_i\subset R^l$ is the control of player i, $u_i$ are piecewise continuous functions. Assume that $h_i\left(u\left(t\right)\right),i=1,\dots ,n$ are differentiable, monotonically increasing and strictly concave.

The system dynamics is described by a first order differential equation:

$$\dot{x}=g(t,u,x),x\left({\tau }_0\right)=x_0,$$

(3)

where $x\in R^r$, (3) satisfy the standard requirements of existence and uniqueness. In particular, we assume that the function $g(t,u,x)$ is continuously differentiable with regards to all its arguments. In addition, as usual for optimal allocation models, $x\left(t\right)\ge 0,$ $g(t,u,x)$ is decreasing on u.

The expected integral payoff of the player i is:

$$K_i(x_0,{\tau }_0,u)=\mathbb{E}\left[\underset{{\tau }_0}{\overset{T}{\int }}{h}_i(x\left(\tau \right),u\left(\tau \right))d\tau \right]=\underset{{\tau }_0}{\overset{\infty }{\int }}\left[\underset{{\tau }_0}{\overset{t}{\int }}{h}_i(x\left(\tau \right),u\left(\tau \right))d\tau \right]dF\left(t\right).$$

(4)

Note that no discount factor appears in the functional objectives of the problem, because the discount structure is built on the characteristics of the random terminal time.

In Section 2, the problem is formulated in general terms, but in Section 4, the functions take on a specific form. This model is chosen to simplify the solution, but the proposed method can, of course, be applied to a wider class of problems.

Assume that players cooperate in order to achieve the maximum total payoff. The optimization problem in this case can be written in the following form:

$$\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^n}{K}_i(x,u)\to \underset{u=(u_1,\dots ,u_n)}{max},\hfill \\ \dot{x}=g(t,x,u),\hfill \\ x\left({\tau }_0\right)=x_0.\hfill \end{array}\right.$$

(5)

This problem was considered by Gromov and Gromova [7] in the class of open-loop strategies, where the control action selects according to the rule $u_i\left(t\right)={\phi }_i\left(t\right)$. Open-loop strategies are usually applied if the players cannot observe the state vector. Here we will restrict our attention to the case when the players apply feedback (or closed-loop) strategies, $u_i\left(t\right)={\phi }_i(x,t)$. This means that players observe the current value of the position $x\left(t\right)$. In general, such structure of strategies is useful in problems in which the players have a chance to react if they observe deviations from some agreed course of players’ action.

According to [19], functional (4) can be simplified to the following form:

$$K_i(x_0,{\tau }_0,u)=\underset{{\tau }_0}{\overset{\infty }{\int }}(1-F\left(\tau \right))h_i(x\left(\tau \right),u_i\left(\tau \right))d\tau .$$

The expected payoff of the player in the subgame ${\mathsf{\Gamma }}^T(x\left(t\right),t)$ starting at the moment t from $x\left(t\right)$ is evaluated by the formula:

$$K_i(t,x\left(t\right),u)=\frac{1}{1-F\left(t\right)}\underset{t}{\overset{\infty }{\int }}(1-F\left(\tau \right))h_i(x\left(\tau \right),u_i\left(\tau \right))d\tau .$$

The optimal cooperative strategies of players $\overline{u}\left(t\right)=({\overline{u}}_1\left(t\right),\dots ,{\overline{u}}_n\left(t\right))$, are defined as follows:

$$\overline{u}\left(t\right)=arg\underset{u_1,\dots ,u_n}{max}\sum _{i=1}^n{K}_i(x,u).$$

The trajectory corresponding to the optimal cooperative strategies $\overline{u}\left(t\right)=({\overline{u}}_1\left(t\right),\dots ,{\overline{u}}_n\left(t\right))$ is the optimal cooperative trajectory $\overline{x}\left(t\right)$.

3. System of Hamilton–Jacobi–Bellman Equations

We will solve the optimization problem using the Hamilton–Jacobi–Bellman equation.

Let $V_j(t,x\left(t\right))$ – be the value of the Bellman function at $t\in [{\tau }_{j-1};{\tau }_j]$:

$$V_j(t,x\left(t\right))=\underset{u}{max}\sum _{i=1}^n{K}_i(t,x\left(t\right),u),t\in [{\tau }_{j-1};{\tau }_j].$$

(6)

Here $\underset{u}{max}{\displaystyle \sum _{i=1}^n}{K}_i(t,x\left(t\right),u)$ – expected maximum total payoff of players in the subgame $\mathsf{\Gamma }(t,x(t\left)\right)$.

Consider subgame $\mathsf{\Gamma }(t,x(t\left)\right)$ starting at the moment $t\ge [{\tau }_{N-1};\infty )$. We have a standard model of the game with random duration here, because there are no switches during the period $({\tau }_{N-1};\infty )$. The expected gain in $\mathsf{\Gamma }(t,x(t\left)\right)$ is as follows:

$$\sum _{i=1}^n{K}_i(t,x\left(t\right),u)=\frac{1}{1-{\overline{F}}_N\left(t\right)}{\int }_t^{+\infty }(1-{\overline{F}}_N\left(\tau \right))\sum _{i=1}^n{h}_i(x\left(\tau \right),u_i\left(\tau \right))\mathrm{d}\tau .$$

In [3], the Hamilton–Jacobi–Bellman equation for differential games with random duration was presented. Then according to the Bellman principle and [3] $V_N(t,x)$ satisfies the equation:

$$\frac{{\overline{f}}_N\left(t\right)}{1-{\overline{F}}_N\left(t\right)}{V}_N(t,x)=\frac{\partial V_N(t,x)}{\partial t}+\underset{u}{max}(h\left(u\right)+\frac{\partial V_N(t,x)}{\partial x}g(x,u))$$

at $t\in [{\tau }_{N-1};\infty )$ and the boundary condition $\underset{t\to \infty }{lim}{V}_N(t,x)=0.$ One can note, that this boundary condition is quite often too strong as a sufficient optimality condition. Weaker conditions that apply to other classes of games can be found in [20,21].

Consider now the subgame $\mathsf{\Gamma }(t,x(t\left)\right)$ at $t\in [{\tau }_{N-2},{\tau }_{N-1}]$. The expected gain is as follows:

$$\begin{array}{c}\frac{1}{1-{\overline{F}}_{N-1}\left(t\right)}({\int }_t^{{\tau }_{N-1}}(1-{\overline{F}}_{N-1}\left(\tau \right))\sum _{i=1}^n{h}_i(x\left(\tau \right),u_i\left(\tau \right))\mathrm{d}\tau +\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\int }_{{\tau }_{N-1}}^{\infty }(1-{\overline{F}}_N\left(\tau \right))\sum _{i=1}^n{h}_i(x\left(\tau \right),u_i\left(\tau \right))\mathrm{d}\tau ).\end{array}$$

(7)

Then, taking in account the solution on the period $[{\tau }_{N-1};\infty )$, and according to [3], $V_{N-1}$ satisfies the equation:

$$\frac{{\overline{f}}_{N-1}\left(t\right)}{1-{\overline{F}}_{N-1}\left(t\right)}{V}_{N-1}=\frac{\partial V_{N-1}}{\partial t}+\underset{u}{max}(h\left(u\right)+\frac{\partial V_{N-1}}{\partial x}g(x,u))$$

at $t\in [{\tau }_{N-2};{\tau }_{N-1})$ and the boundary condition

$$V_{N-1}({\tau }_{N-1},x\left({\tau }_{N-1}\right))=V_N({\tau }_{N-1},x\left({\tau }_{N-1}\right)).$$

Increasing the range of t values step by step, we arrive at the Bellman function describing the maximum gain on $[{\tau }_0,{\tau }_N]$.

So, for $j=1,\dots ,N-1$ we have:

$$\left\{\begin{array}{cc}\frac{{\overline{f}}_j\left(t\right)}{1-{\overline{F}}_j\left(t\right)}{V}_j(t,x)=\frac{\partial V_j(t,x)}{\partial t}+\underset{u}{max}(h\left(u\right)+\frac{\partial V_j(t,x)}{\partial x}g(x,u)),\hfill & t\in [{\tau }_{j-1};{\tau }_j],\hfill \\ V_j({\tau }_j,x\left({\tau }_j\right))=V_{j+1}({\tau }_j,x\left({\tau }_j\right)).\hfill \end{array}\right.$$

(8)

4. Example

Consider the game ${\mathsf{\Gamma }}^T(x_0,{\tau }_0)$ of non-renewable resource extraction by two players with instantaneous payoffs $h_i\left(u_i\right)=\sqrt{u_i\left(t\right)}$. Let $h\left(u\right)={\displaystyle \sum _{i=1}^2}{h}_i(x,u_i)$. Suppose the players use identical equipment to operate the field and the probability of equipment failure is determined by the modes of operation. On this model starting at the time instant ${\tau }_0=0$ from the initial state $x_0$, $u_i\left(t\right)$ denotes the extraction effort of player i at time t. The system dynamics $x\left(t\right)$ (the resource stock at time t available to the extraction) is described by the first order differential equation:

$$\dot{x}=-\sum _{i=1}^2{u}_i,x\left(0\right)=x_0.$$

We have the following problem of optimal control:

$$\sum _{i=1}^n{\int }_{{\tau }_0}^{{\tau }_N}(1-F\left(\tau \right))\sqrt{u_i\left(t\right)}d\tau \to \underset{u=(u_1,u_2)}{max}.$$

4.1. Time-Dependent Switch

In this example, the duration of the game T is a random variable with composite distribution function $F\left(t\right)$ defined by the rule (9):

$$F\left(t\right)=\left\{\begin{array}{cc}0,\hfill & t\in (-\infty ,0),\hfill \\ F_1\left(t\right),\hfill & t\in [0,{\tau }_1),\hfill \\ {\alpha }_1\left({\tau }_1\right)F_2\left(t\right)+{\beta }_1\left({\tau }_1\right),\hfill & t\in [{\tau }_1,\infty ).\hfill \end{array}\right.$$

(9)

Here, we assume that ${\tau }_1$ is a moment of switching of distribution function in this game. Let there be an ordered sequence of time moments $\tau :0={\tau }_0<{\tau }_1<{\tau }_2=\infty$. The optimal control problem can be decomposed into two problems, on the intervals $I_1=[0,{\tau }_1]$, $I_2=[{\tau }_1,\infty )$.

The distribution functions $F_j\left(t\right),j=1,2$ are defined using an exponential distribution:

$$F_1\left(t\right)=1-e^{-{\lambda }_{1}t},t\in [0,{\tau }_1),$$

$$F_2\left(t\right)=1-e^{-{\lambda }_{2}t},t\in [{\tau }_1,\infty ).$$

Here, ${\lambda }_1,{\lambda }_2$—parameters of exponential distributions, ${\lambda }_1,{\lambda }_2>0$.

Let us find ${\alpha }_1\left({\tau }_1\right)$ and ${\beta }_1\left({\tau }_1\right)$:

$${\alpha }_1\left({\tau }_1\right)=\frac{F\left({\tau }_1^{-}\right)-1}{F_2\left({\tau }_1\right)-1}=\frac{F_1\left({\tau }_1\right)-1}{F_2\left({\tau }_1\right)-1}{F}_2\left(t\right),$$

$${\beta }_1\left({\tau }_1\right)=1-\frac{F\left({\tau }_1^{-}\right)-1}{F_2\left({\tau }_1\right)-1}=1-\frac{F_1\left({\tau }_1\right)-1}{F_2\left({\tau }_1\right)-1}{F}_2\left(t\right),$$

$${\alpha }_1\left({\tau }_1\right)F_2\left(t\right)+{\beta }_1\left({\tau }_1\right)=1-\frac{e^{-{\lambda }_1{\tau }_1}}{e^{-{\lambda }_2{\tau }_1}}{e}^{-{\lambda }_{2}t},t\in [{\tau }_1,\infty ).$$

Then, the composite distribution function for one switch at a fixed moment in time can be written in the form:

$$F\left(t\right)=\left\{\begin{array}{cc}0,\hfill & t\in (-\infty ,0),\hfill \\ 1-e^{-{\lambda }_{1}t},\hfill & t\in [0,{\tau }_1),\hfill \\ 1-e^{-{\lambda }_{2}t}{e}^{{\tau }_1({\lambda }_2-{\lambda }_1)},\hfill & t\in [{\tau }_1,\infty ).\hfill \end{array}\right.$$

(10)

Figure 1 shows the example of such function.

One reason for using a composite distribution function in such models could be that as a non-renewable resource is mined, the remaining resource may become increasingly difficult to extract. Which leads to another way of computing the probability of the end of the process. On the other hand the probability of a breakdown may depend on the development stage. It may be higher in the beginning of the process.

Consider a subgame $\mathsf{\Gamma }(t,x(t\left)\right)$, when $t\ge {\tau }_1$. We need to solve the Hamilton–Jacobi–Bellman equation for this problem:

$$\frac{{\overline{f}}_2\left(t\right)}{1-{\overline{F}}_2\left(t\right)}{V}_2(t,x)=\frac{\partial V_2(t,x)}{\partial t}+\underset{u}{max}(h\left(u\right)+\frac{\partial V_2(t,x)}{\partial x}(-u_1-u_2)),$$

(11)

with the boundary condition

$$\underset{t\to \infty }{lim}{V}_2(t,x)=0.$$

Here

$$\frac{{\overline{f}}_2\left(t\right)}{1-{\overline{F}}_2\left(t\right)}={\lambda }_2\frac{e^{-{\lambda }_1{\tau }_1}{e}^{-{\lambda }_{2}t}{e}^{-{\lambda }_2{\tau }_1}}{e^{-{\lambda }_2{\tau }_1}{e}^{-{\lambda }_1{\tau }_1}{e}^{-{\lambda }_{2}t}}={\lambda }_2.$$

The solution of (11) is found in the form

$$V_2(t,x)=a\left(t\right)\sqrt{x}.$$

The maximization problem in (11) yields a strategy for player i:

$$u_i\left(t\right)=\frac{1}{2\frac{\partial V_2}{\partial x}}=\frac{x}{a^2\left(t\right)},i=1,2.$$

Substituting it into (11), we have the following differential equation for $a\left(t\right)$:

$${\lambda }_{2}a\left(t\right)-\dot{a}\left(t\right)=\frac{1}{a\left(t\right)}.$$

(12)

The solution of (12) is

$$a\left(t\right)=\sqrt{\frac{e^{2{\lambda }_{2}t}C+1}{{\lambda }_2}}.$$

The form of optimal control is as follows:

$$u_i=\frac{x}{a^2}=\frac{{\lambda }_{2}x}{e^{2{\lambda }_{2}t}C+1}.$$

Denote $x\left({\tau }_1\right)=x_1$. Now, solving the first order differential equation

$$\dot{x}=-u_1-u_2$$

with initial condition $x\left({\tau }_1\right)=x_1$, we obtain:

$$x\left(t\right)=\frac{x_1(e^{-2{\lambda }_{2}t}+C)}{e^{-2{\lambda }_2{\tau }_1}+C}.$$

According to the boundary condition $\underset{t\to \infty }{lim}{V}_2(t,x)=0$, we obtain $C=0$, then the solution on the interval $I_2$ takes the form:

$${\overline{u}}_i={\lambda }_{2}x,$$

$$\overline{x}=x_1{e}^{2{\lambda }_2({\tau }_1-t)}.$$

Consider a subgame $\mathsf{\Gamma }(t,x(t\left)\right)$, when $t<{\tau }_1$. Let $V_1(t,x)$ be a Bellman function on this interval.

The Hamilton–Jacobi–Bellman equation for this problem with a random moment of game ending:

$$\frac{f_1\left(t\right)}{1-F_1\left(t\right)}{V}_1=\frac{\partial V_1}{\partial t}+\underset{u}{max}(h\left(u\right)+\frac{\partial V_1}{\partial x}(-u_1-u_2)),$$

(13)

with the boundary condition

$$V_1({\tau }_1,x\left({\tau }_1\right))=a\left({\tau }_1\right)\sqrt{x\left({\tau }_1\right)}.$$

We will look for the Bellman function in the form $V_1(t,x)=\tilde{a}\left(t\right)\sqrt{x}$. Then according to the boundary condition $\tilde{a}\left({\tau }_1\right)=a\left({\tau }_1\right)=\frac{1}{\sqrt{{\lambda }_2}}$.

Note that

$$\frac{f_1\left(t\right)}{1-F_1\left(t\right)}=\frac{{\lambda }_1{e}^{-{\lambda }_{1}t}}{e^{-{\lambda }_{1}t}}={\lambda }_1.$$

The maximization problem in (13) yields a strategy for player i:

$$u_i\left(t\right)=\frac{x}{{\tilde{a}}^2\left(t\right)},i=1,2.$$

Substituting it into (13), we have the following differential equation for $\tilde{a}\left(t\right)$:

$${\lambda }_1\tilde{a}\left(t\right)-\dot{\tilde{a}}\left(t\right)=\frac{1}{\tilde{a}\left(t\right)},\tilde{a}\left({\tau }_1\right)=\frac{1}{\sqrt{{\lambda }_2}}.$$

(14)

The solution of (14) is

$$\tilde{a}\left(t\right)=\sqrt{\frac{e^{2{\lambda }_1(t-{\tau }_1)}({\lambda }_1-{\lambda }_2)+{\lambda }_2}{{\lambda }_1{\lambda }_2}}.$$

The form of control is as follows:

$$u_i=\frac{x}{{\tilde{a}}^2}=\frac{{\lambda }_1{\lambda }_{2}x}{e^{2{\lambda }_1(t-{\tau }_1)}({\lambda }_1-{\lambda }_2)+{\lambda }_2}.$$

Solving the first order differential equation

$$\dot{x}=-u_1-u_2,$$

with initial condition $x\left(0\right)=x_0$, we obtain:

$$\overline{x}\left(t\right)=x_0\frac{{\lambda }_2(e^{2{\lambda }_1({\tau }_1-t)}-1)+{\lambda }_1}{{\lambda }_2(e^{2{\lambda }_1{\tau }_1}-1)+{\lambda }_1}.$$

To sum up, the cooperative trajectory $\overline{x}\left(t\right)$ and optimal cooperative controls ${\overline{u}}_i\left(t\right)$ at intervals $I_1$ and $I_2$ have the form:

$$\overline{x}\left(t\right)=\left\{\begin{array}{cc}{x}_0\frac{{\lambda }_2(e^{2{\lambda }_1({\tau }_1-t)}-1)+{\lambda }_1}{{\lambda }_2(e^{2{\lambda }_1{\tau }_1}-1)+{\lambda }_1}\hfill & t\in [0,{\tau }_1),\hfill \\ \frac{x_0{\lambda }_1{e}^{2{\lambda }_2({\tau }_1-t)}}{{\lambda }_2(e^{2{\lambda }_1{\tau }_1}-1)+{\lambda }_1},\hfill & t\in [{\tau }_1,\infty ).\hfill \end{array}\right.$$

(15)

$${\overline{u}}_i(t,x)=\left\{\begin{array}{cc}\frac{x{\lambda }_2{\lambda }_1}{e^{2{\lambda }_1(t-{\tau }_1)}({\lambda }_1-{\lambda }_2)+{\lambda }_2},\hfill & t\in [0,{\tau }_1),\hfill \\ {\lambda }_{2}x,\hfill & t\in [{\tau }_1,\infty ).\hfill \end{array}\right.$$

(16)

Finally, we show the graphical representation of the obtained solutions. We assume the following values of the parameters: ${\lambda }_1=0.01$, ${\lambda }_2=0.1$, $x_0=100$, ${\tau }_1=40$. Figure 2 shows the optimal control of player 1 (the blue line corresponds to $t\in [0,{\tau }_1]$, the red line for $t\in [{\tau }_1,\infty )$). The optimal cooperative trajectory $\overline{x}\left(t\right)$ is presented by Figure 3.

4.2. State-Dependent Switch

Let us assume now that the stock level of the resource can influence the probability of a regime shift, so the switching occurs not at some fixed point in time, but when a certain condition on the trajectory is satisfied. Within the framework of the model under consideration, such a condition may be the achievement of some predetermined level of resource stock.

Let the switching time ${\tau }_1$ is determined from the condition $x\left({\tau }_1\right)=x_1$, where $x_0>x_1>0$, $x_1$ is fixed.

On the interval $I_2=[{\tau }_1,\infty )$ the solution of Hamilton–Jacobi–Bellman Equation (11) is the same as in the previous case:

$$V_2(t,x)=a\left(t\right)\sqrt{x},$$

$${\overline{u}}_i={\lambda }_{2}x,$$

$$\overline{x}=x_1{e}^{2{\lambda }_2({\tau }_1-t)}.$$

On the interval $I_1=[{\tau }_0,{\tau }_1]$ since $x_1$ is fixed we have a new condition on the trajectory that must be satisfied $x\left({\tau }_1\right)=x_1$.

Here, the solution of (13) is found in the form

$$V_1(t,x)=c\left(t\right)\sqrt{x}+b\left(t\right).$$

The maximization problem in (13) yields a strategy for player i:

$$u_i\left(t\right)=\frac{x}{c^2\left(t\right)},i=1,2.$$

Substituting it into (13), we have the following differential equations for $c\left(t\right)$ and $b\left(t\right)$:

$$\begin{array}{c}{\lambda }_{1}c\left(t\right)-\dot{c}\left(t\right)=\frac{1}{c\left(t\right)},\\ \dot{b}\left(t\right)={\lambda }_{1}b\end{array}$$

(17)

with the boundary condition

$$c\left({\tau }_1\right)\sqrt{x_1}+b\left({\tau }_1\right)=a\left({\tau }_1\right)\sqrt{x_1}=\frac{\sqrt{x_1}}{{\lambda }_2}.$$

(18)

The solution of (17) is

$$c\left(t\right)=\sqrt{\frac{e^{2{\lambda }_{1}t}{C}_1+1}{{\lambda }_1}},$$

$$b\left(t\right)=C_2{e}^{{\lambda }_{1}t}.$$

The form of control is as follows:

$$u_i=\frac{x}{c^2}=\frac{{\lambda }_{1}x}{e^{2{\lambda }_{1}t}{C}_1+1}.$$

Solving the first order differential equation

$$\dot{x}=-u_1-u_2$$

we obtain:

$$\overline{x}\left(t\right)=(e^{-2{\lambda }_{1}t}+C_1)C_3.$$

Taking into account the boundary conditions on the trajectory $x\left(0\right)=x_0$, $x\left({\tau }_1\right)=x_1$, we obtain:

$$C_1=\frac{x_1-x_0{e}^{-2{\lambda }_1{\tau }_1}}{x_0-x_1},$$

$$C_3=\frac{x_0-x_1}{1-e^{-2{\lambda }_1{\tau }_1}}.$$

Finally, from (18) we have

$$C_2=\sqrt{x_1}{e}^{-{\lambda }_1{\tau }_1}\left(\frac{1}{\sqrt{{\lambda }_2}}-\sqrt{\frac{x_1(e^{2{\lambda }_1{\tau }_1}-1)}{(x_0-x_1){\lambda }_1}}\right).$$

Then, the maximum total payoff of players in the game $\mathsf{\Gamma }({\tau }_0,x_0)$ has the following form:

$$V_1(0,x_0)=\sqrt{\frac{(x_0-x_1)(1-e^{-2{\lambda }_1{\tau }_1})}{{\lambda }_1}}+\frac{x_1}{{\lambda }_2}{e}^{-{\lambda }_1{\tau }_1}.$$

The next problem is finding the optimal switching time ${\tau }_1^{*}$:

$${\tau }_1^{*}=arg\underset{{\tau }_1}{max}{V}_1(0,x_0)=\frac{1}{2{\lambda }_1}ln(1+\frac{{\lambda }_2(x_0-x_1)}{{\lambda }_1{x}_1}).$$

Figure 4 demonstrates the graph of $F\left(t\right)$ in our example if ${\tau }_1^{*}$ is a switching time and $x_1=20$. This value means that when 80% of the initial stock of the resource is used up, a switch to another mode will occur. With the model parameters we have chosen (${\lambda }_1=0.01$, ${\lambda }_2=0.1$), it turns out that after switching, the probability of ending the game increases.

To sum up, the cooperative trajectory $\overline{x}\left(t\right)$ and optimal cooperative controls ${\overline{u}}_i\left(t\right)$ at intervals $I_1$ and $I_2$ have the form:

$$\overline{x}\left(t\right)=\left\{\begin{array}{cc}\frac{x_0(e^{-2{\lambda }_{1}t}-e^{-2{\lambda }_1{\tau }_1})+x_1(1-e^{-2{\lambda }_{1}t})}{1-e^{-2{\lambda }_1{\tau }_1}}\hfill & t\in [0,{\tau }_1^{*}),\hfill \\ x_1{e}^{2{\lambda }_2({\tau }_1-t)},\hfill & t\in [{\tau }_1^{*},\infty ).\hfill \end{array}\right.$$

(19)

$${\overline{u}}_i(t,x)=\left\{\begin{array}{cc}\frac{{\lambda }_1(x_0-x_1)x}{x_1(e^{2{\lambda }_{1}t}-1)+x_0(1-e^{-2{\lambda }_1({\tau }_1-t)})},\hfill & t\in [0,{\tau }_1^{*}),\hfill \\ {\lambda }_{2}x,\hfill & t\in [{\tau }_1^{*},\infty ).\hfill \end{array}\right.$$

(20)

Figure 5 and Figure 6 demonstrate graphs of optimal control and optimal cooperative trajectory for state-dependent case. Here, ${\lambda }_1=0.01$, ${\lambda }_2=0.1$, $x_0=100$, $x_1=20$.

Note that in both cases, after switching the cumulative distribution function, players increase the intensity of resource extraction. This is because in our example ${\lambda }_2>{\lambda }_1$, so the probability of ending the game increases in the second interval.

5. Conclusions

A method of the construction of optimal closed-loop cooperative strategies in differential games with random duration and composite distribution function of terminal time is proposed. Due to the switching modes of the distribution function, the payoffs of players in such games take the form of the sums of integrals with different, but adjoint time intervals. A method for solving such problems using dynamic programming is explored. It is shown that finding the optimal control in such problems is reduced to the sequential consideration of intervals, starting from the last switching, and compiling the Hamilton–Jacobi–Bellman equation on each interval, and boundary solutions are obtained from the solution on the interval considered earlier. An example of a non-renewable resource extraction model is studied. Two cases are considered. In the first one, the switching of the terminal time distribution function occurs at a predetermined time. In the second, the switching depends on the state of the system. An analytical view of the optimal control and the optimal cooperative trajectory is obtained. It was demonstrated how the optimal controls of the players changes when switching the distribution function.

The future research will consist in studying the non-cooperative case of the game and time-consistency problem for obtained solutions.