Subgame Consistent Cooperative Behavior in an Extensive form Game with Chance Moves

Abstract

We design a mechanism of the players’ sustainable cooperation in multistage n-person game in the extensive form with chance moves. When the players agreed to cooperate in a dynamic game they have to ensure time consistency of the long-term cooperative agreement. We provide the players’ rank based (PRB) algorithm for choosing a unique cooperative strategy profile and prove that corresponding optimal bundle of cooperative strategies satisfies time consistency, that is, at every subgame along the optimal game evolution a part of each original cooperative trajectory belongs to the subgame optimal bundle. We propose a refinement of the backwards induction procedure based on the players’ attitude vectors to find a unique subgame perfect equilibrium and use this algorithm to calculate a characteristic function. Finally, to ensure the sustainability of the cooperative agreement in a multistage game we employ the imputation distribution procedure (IDP) based approach, that is, we design an appropriate payment schedule to redistribute each player’s optimal payoff along the optimal bundle of cooperative trajectories. We extend the subgame consistency notion to extensive-form games with chance moves and prove that incremental IDP satisfies subgame consistency, subgame efficiency and balance condition. An example of a 3-person multistage game is provided to illustrate the proposed cooperation mechanism.

1. Introduction

In a dynamic n-person game the players first choose their “optimal” strategies at the initial position $x_0$ (which form the optimal strategy profile for the whole game), and then have an option to change their strategies at any intermediate position $x_t$ and switch to other strategies if these strategies constitute the locally optimal strategy profile for the subgame starting at $x_t$. The time consistency property (first introduced in References [1,2,3] for differential games) ensures that the players will not have an incentive to change their strategies at any subgame along the optimal game evolution, and hence plays an important role in the designing of the optimal players’ behavior in non-cooperative and cooperative dynamic games (see, e.g., References [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21], for details).

We consider an n-person finite multistage games in the extensive form (see, e.g., References [5,17,22,23]) with perfect information and with chance moves. Note that much research has been already done on time consistent solutions (or close concepts) in extensive-form games (see, e.g., References [4,6,13,17,21]). Time consistency concept was extended to dynamic games played over event trees in References [14,16,20] as well as to multicriteria extensive-form cooperative games (without chance moves) in References [7,8,10,11,15]. The property of "time consistency in the whole game" was extended to multicriteria extensive-form cooperative games with chance moves in Reference [9] (note that in these games an optimal pure strategy profile does not generate the unique optimal trajectory in the game tree but rather the whole optimal bundle of the trajectories).

In the paper, we mainly focus on the dynamic aspects of cooperation in a dynamic extensive-form game with chance moves, and propose to design a mechanism of the players’ sustainable cooperation which satisfies three properties. First, a fragment of each cooperative trajectory from the optimal bundle for the original game ${\mathrm{\Gamma }}^{x_0}$ should “remain optimal” at each subgame ${\mathrm{\Gamma }}^{x_t}$ along the cooperative game evolution, that is, it should belong to the subgame optimal bundle of cooperative trajectories. Secondarily, a cooperative payoff-to-go at the subgame ${\mathrm{\Gamma }}^{x_t}$ is no less than the non-cooperative payoff-to-go for all players. Finally, when the players re-evaluate their expected cooperative payoff after each passed chance move, they have no incentive to change original cooperative agreement.

To this aim, we first need to provide a rule for choosing a unique cooperative strategy profile as well as the unique optimal bundle of cooperative trajectories. We introduce the Players’ Rank Based (PRB) algorithm and prove that this algorithm generates the unique optimal bundle of cooperative trajectories which satisfies time consistency. Note that a rather close approach—the so-called Refined Leximin (RL) algorithm—was introduced recently in Reference [8]. Let us notice the main differences of these two algorithms. The RL algorithm is applicable for multicriteria game without chance moves and is based on the ranking of the criteria, while the PRB algorithm is designed for single-criterion extensive-form game with chance moves and employed the players’ ranks. Further, the RL algorithm allows to choose a unique cooperative trajectory while the PRB algorithm generates the unique optimal bundle of the cooperative trajectories in the game tree. To the best of the authors’ knowledge, other approaches to choosing an optimal bundle of the cooperative trajectories in extensive-form game with chance moves have not been considered yet.

Then, to construct a characteristic function (which describes the worth of each coalition in cooperative game) we use an equilibrium-based approach, namely the $\gamma$-characteristic function introduced in Reference [24]. Hence, the players have to accept a specific method for choosing a unique Subgame Perfect Equilibria (SPE) [25] in an extensive-form game with chance moves. To solve this problem we provide the novel refinement of the backwards induction procedure (see, e.g., References [5,17,23])—the so-called Attitude SPE algorithm. A similar approach to construct a unique SPE in extensive-form game with perfect information was explored in References [17,26,27] and was called the Type Equilibrium (TE) algorithm. Both algorithms are the refinements of the general backwards induction procedure that take into account the attitudes of each player towards other players. Let us point out the main differences of these algorithms. The TE algorithm is applicable for the game without chance moves and for the case when the payoffs are only determined in terminal nodes. In addition, the TE algorithm allows to construct SPE that is “unique” in the sense of payoffs (i.e., there may exist several optimal trajectories which generate the same equilibrium payoffs) while the Attitude SPE algorithm allows to choose unique SPE strategy profile as well as unique bundle of trajectories. Another rather close approach to find a unique SPE—the so-called Indifferent Equilibrium (IE) algorithm—was introduced in Reference [28]. Again, the IE algorithm is applicable only for the game without chance moves and for the partial case when the payoffs are determined in terminal nodes. Moreover, IE algorithm in general allows to construct a SPE in behavior strategies while the proposed Attitude SPE algorithm always generates a SPE in pure strategies.

It is worth noting, that other approaches to analyze an extensive-form game, except for the backwards induction procedure and its refinements mentioned above, imply that the researcher first needs to obtain a strategic representation of the original extensive game and then analyzes this strategic (or normal-form) game (see, e.g., References [29,30,31] ). For instance, the software tool “Game Theory Explorer” [29] is based on the strategic-form representation and then applying the modified Lemke-Howson algorithm [32] to find all Nash equilibria. The majority of existing algorithms are developed to find Nash equilibria in mixed strategies for 2-person games and do not allow to construct SPE in pure strategies. Moreover, as it was noted in Reference [31], in general the strategic-form representation is exponential in the size of the original game tree. In contrast, the proposed Attitude SPE algorithm is a rather simple recursive algorithm which deals with n-person extensive-form game (with perfect information) itself and allows to compute a unique SPE in pure strategies.

After computing the $\gamma$-characteristic function we suppose that the players adopt some single-valued cooperative solution $\phi$ (for instance, the Shapley value [33], the nucleolus [34], etc.) which satisfies the individual and collective rationality property. Finally, to guarantee the sustainability of the achieved long-term cooperative agreement we employ the Imputation Distribution Procedure (IDP) based approach (see, e.g., References [3,12,14,16,17,18,20,35]), that is, a payment schedule to redistribute the ith player’s expected cooperative payoff along the optimal bundle of cooperative trajectories. In this paper, we mainly focus on the following good properties an IDP may satisfy: subgame efficiency, strict balance condition [10,15,17] and an appropriate refinement of the time consistency property, called subgame consistency. The point is that the “time consistency in the whole game” property [9,14,16,20] is based on an a priori assessment of the ith player’s expected optimal payoff (before the game ${\mathrm{\Gamma }}^{x_0}$ starts). However, when the players make a decision in the subgame ${\mathrm{\Gamma }}^{x_t}$ after the chance move occurs, they need to re-estimate their expected optimal payoffs-to-go since the original optimal bundle of cooperative trajectories shrinks after each chance node. To deal with this interesting feature of the game with chance moves we adopt the notion of subgame consistency that was firstly proposed in Reference [36] for cooperative stochastic differential games and then extend it to stochastic dynamic games in References [37,38].

Since we derive a suitable definition of subgame consistency for other class of games, the proposed Definition 6 differs from ones provided in References [37,38] but captures the same idea. Let us point out the main differences with References [37,38]. While D. Yeung and L. Petrosyan do not consider the issue of multiple equilibria and study the stochastic games in which there exists a unique Nash equilibrium in each subgame, we focus on the problem of how to select a unique (subgame perfect) Nash equilibrium in extensive-form game with chance moves and derive the corresponding algorithm. Secondarily, the characteristic function has not been constructed in References [37,38] and, hence, the players are restricted to using the simplest cooperative solutions (for instance, they may share equally the excess of the total expected cooperative payoff over the expected sum of individual non-cooperative payoffs), whereas we provide a method for calculating the $\gamma$-characteristic function. Hence, the players may use different solution concepts based on the characteristic function approach. Finally, it turns out that the incremental IDP specified for extensive-form games with chance moves in Reference [9] satisfies not only the subgame consistency but also subgame efficiency and strict balance condition.

Therefore, the suggested PRB algorithm, the Attitude SPE algorithm combined with the $\gamma$-characteristic function, and the incremental payment schedule for any single-valued cooperative solution (meeting individual and collective rationality) together constitute a required mechanism of the players’ sustainable cooperation that satisfies exactly three properties mentioned above for any extensive-form game with chance moves.

It is worth noting that the extensive-form games, as well as dynamic games played over event trees, differential games and multistage games with discrete dynamics are used to model various real-world situations where several decision makers (or players) with different objectives may cooperate (see, e.g., References [5,12,14,16,17,20,39,40,41,42,43,44]. Hence, a proposed approach to implement a long-term cooperative agreement may have a number of possible applications.

The rest of the paper is organized as follows: Section 2 recalls the main ingredients of the class of games of interest. In Section 3, we specify the attitude SPE algorithm that allows constructing a unique SPE in a extensive-form game with chance moves. In Section 4, we provide the PRB algorithm and prove that the optimal bundle of cooperative trajectories generated by this algorithm satisfy time consistency. Section 5 reveals a drawback of the IDP “time consistency in the whole game” property and presents a subgame consistency definition that is applicable for extensive-form games with chance moves. We prove that incremental IDP satisfies a number of good properties and consider an example of a 3-person multistage game with chance moves to illustrate the incremental IDP implementation. Section 6 provides a brief review of the results and discussion.

2. Extensive-Form Game with Chance Moves

We consider a finite multistage game in extensive form following References [6,13,17,22,23]. First we need to define the basic notations and briefly remind some properties of extensive-form game that will be used in the sequel:

• $N=\{1,\dots ,n\}$ is the set of all players.
• K is the game tree with the root $x_0$ and the set of all nodes P.
• $S\left(x\right)$ is the set of all direct successors (descendants) of the node x, and $S^{-1}\left(y\right)$ is the unique predecessor (parent) of the node $y\ne x_0$ such that $y\in S\left(S^{-1}\left(y\right)\right)$.
• $P_i$ is the set of all decision nodes of the ith player (at these nodes the player i chooses the following node), $P_i\cap P_j=\varnothing$, for all $i,j\in N$, $i\ne j$.
• $P_{n+1}={\left\{z^j\right\}}_{j=1}^m$ denotes the set of all terminal nodes (final positions), $S\left(z^j\right)=\varnothing$$\forall z^j\in P_{n+1}$.
• $P_0$ is the set of all nodes at which a chance moves, where $\pi \left(y\right|x)>0$ denotes the probability of transition from node $x\in P_0$ to node $y\in S\left(x\right)$. We suppose that for each $x\in P_0$ it holds that $S\left(x\right)\cap P_0=\varnothing$. Lastly, ${\displaystyle \bigcup _{i=0}^{n+1}}{P}_i=P$.
• $\omega =(x_0,\dots ,x_{t-1},x_t,\dots ,x_T)$ is the trajectory (or the path) in the game tree, $x_{t-1}=S^{-1}\left(x_t\right),1\le t\le T$, $x_T=z^j\in P_{n+1}$; where index t in $x_t$ denotes the ordinal number of this node within the trajectory $\omega$ and can be interpreted as the "time index".
• $h_i\left(x\right)=(h_{i/1}\left(x\right),\dots ,h_{i/r}\left(x\right))$ is the payoff of the ith player at the node $x\in P$. We assume that for all $i\in N$, $k=1,\dots ,r$, and $x\in P$ the payoffs are non-negative, that is, $h_{i/k}\left(x\right)⩾0$.

In the following, we will use $G^{cm}\left(n\right)$ to denote the class of all finite multistage n-person games with chance moves in extensive form defined above, where ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$ denotes a game with root $x_0$. Note that ${\mathrm{\Gamma }}^{x_0}$ is an extensive-form game with perfect information (see, e.g., References [17,22,23] for details).

Since all the solutions we are interested in throughout the paper are attainable when the players restrict themselves to the class of pure strategies we will focus on this class of strategies. The pure strategy $u_i(·)$ of the ith player is a function with domain $P_i$ that specifies for each node $x\in P_i$ the next node $u_i\left(x\right)\in S\left(x\right)$ which the player i has to choose at x. Let $U_i$ denote the (finite) set of all ith player’s pure strategies, $U={\prod }_{i\in N}{U}_i$.

Denote by $p\left(y\right|x,u)$ the conditional probability that node $y\in S\left(x\right)$ is reached if node x has been already reached (the probability of transition from x to y) while the players use the strategies $u_i,i\in N$. Note that for all $x\in P_i$, $i=1,\cdots ,n$, and for all $y\in S\left(x\right)$$p\left(y\right|x,u)=1$ if $u_i\left(x\right)=y$ and $p\left(y\right|x,u)=0$ if $u_i\left(x\right)\ne y$. For chance moves, that is, if $x\in P_0$$p\left(y\right|x,u)=\pi (y\left|x\right)$ for all $y\in S\left(x\right)$ for each $u\in U$.

Then one can calculate the probability $p(\omega ,u)$ of realization of the trajectory $\omega =(x_0,\dots ,x_{\tau },x_{\tau +1},\dots ,x_T)$, $x_T\in P_{n+1}$, $x_{\tau +1}\in S\left(x_{\tau }\right)$, $\tau =0,\dots ,T-1$, when the players use the strategies $u_i$ from the strategy profile $u=(u_1,\dots ,u_n)$.

$$p(\omega ,u)=p\left(x_1\right|x_0,u)·p\left(x_2\right|x_1,u)·\dots ·p\left(x_T\right|x_{T-1},u)=\prod _{\tau =0}^{T-1}p\left(x_{\tau +1}\right|x_{\tau },u).$$

(1)

Denote by $\mathrm{\Omega }\left(u\right)=\left\{{\omega }_k\left(u\right)\right|p({\omega }_k,u)>0\}$ the finite set (or the bundle) of the trajectories ${\omega }_k$ which are generated by strategy profile $u\in U$. Note that for all ${\omega }_k\left(u\right)\in \mathrm{\Omega }\left(u\right)$, $u_j\left(x_{\tau }\right)=x_{\tau +1}$ for all $x_{\tau }\in {\omega }_k\left(u\right)\cap P_j$, $j\in N$, $0\le \tau \le T-1$.

Let ${\tilde{h}}_i\left(\omega \right)={\displaystyle \sum _{\tau =0}^T}{h}_i\left(x_{\tau }\right)$ denote the ith player’s vector payoff corresponding to the trajectory $\omega =(x_0,\dots ,x_t,x_{t+1},\dots ,x_T)$.

Denote by

$$H_i\left(u\right)={\displaystyle \sum _{{\omega }_k\in \mathrm{\Omega }\left(u\right)}}p({\omega }_k,u)·{\tilde{h}}_i\left({\omega }_k\right)={\displaystyle \sum _{{\omega }_k\in \mathrm{\Omega }\left(u\right)}}p({\omega }_k,u)·\sum _{\tau =0}^{T\left(k\right)}{h}_i\left(x_{\tau }\right)$$

(2)

the (expected) value of the ith player’s payoff function which corresponds to the strategy profile $u=(u_1,\dots ,u_n)$. Let ${\mathrm{\Omega }}_{n+1}\left(u\right)=\{\mathrm{\Omega }\left(u\right)\cap P_{n+1}\}$ denote the set of all terminal nodes of the trajectories ${\omega }_k\left(u\right)\in \mathrm{\Omega }\left(u\right)$.

Remark 1

([9]). If the pure strategy profiles u and v generate different bundles $\mathrm{\Omega }\left(u\right)$ and $\mathrm{\Omega }\left(v\right)$ of the trajectories, that is, $\mathrm{\Omega }\left(u\right)\ne \mathrm{\Omega }\left(v\right)$, then ${\mathrm{\Omega }}_{n+1}\left(u\right)\cap {\mathrm{\Omega }}_{n+1}\left(v\right)=\varnothing$.

According to References [17,22,23] each intermediate node $x_t\in P\backslash P_{n+1}$ generates a subgame ${\mathrm{\Gamma }}^{x_t}$ with the subgame tree $K^{x_t}$ and the subgame root $x_t$ as well as a factor-game ${\mathrm{\Gamma }}^D$ with the factor-game tree $K^D=(K\backslash K^{x_t})\cup \left\{x_t\right\}$. Decomposition of the original extensive game ${\mathrm{\Gamma }}^{x_t}$ at node $x_t$ onto the subgame ${\mathrm{\Gamma }}^{x_t}$ and the factor-game ${\mathrm{\Gamma }}^D$ generates the corresponding decomposition of the pure (and mixed) strategies (see References [17,22] for details).

Let $P_i^{x_t}\left(P_i^D\right)$, $i\in N$, denote the restriction of $P_i$ on the subgame tree $K^{x_t}\left(K^D\right)$, and $u_i^{x_t}\left(u_i^D\right)$, $i\in N$, denote the restriction of the ith player’s pure strategy $u_i(·)$ in ${\mathrm{\Gamma }}^{x_0}$ on $P_i^{x_t}\left(P_i^D\right)$. The pure strategy profile $u^{x_t}=(u_1^{x_t},\dots ,u_n^{x_t})$ generates the bundle of the subgame trajectories ${\mathrm{\Omega }}^{x_t}\left(u^{x_t}\right)=\left\{{\omega }_k^{x_t}\left(u^{x_t}\right)\right|p({\omega }_k^{x_t},u^{x_t})$ $>0\}.$ Similarly to (2), let us denote by

$$H_i^{x_t}\left(u^{x_t}\right)={\displaystyle \sum _{{\omega }_k^{x_t}\in {\mathrm{\Omega }}^{x_t}\left(u^{x_t}\right)}}p({\omega }_k^{x_t},u^{x_t})·\sum _{\tau =t}^{T\left(k\right)}{h}_i\left(x_{\tau }\right)={\displaystyle \sum _{{\omega }_k^{x_t}\in {\mathrm{\Omega }}^{x_t}\left(u^{x_t}\right)}}p({\omega }_k^{x_t},u^{x_t})·{\tilde{h}}_i\left({\omega }_k^{x_t}\right)$$

(3)

the expected value of the ith player’s payoff in ${\mathrm{\Gamma }}^{x_t}$, and by $U_i^{x_t}$ the set of all possible ith player’s pure strategies in the subgame ${\mathrm{\Gamma }}^{x_t}$, $U^{x_t}={\displaystyle \prod _{i\in N}}{U}_i^{x_t}$. Note that for each trajectory $\omega =(x_0,\dots ,x_t,x_{t+1},\dots ,x_T)$, $1⩽t⩽T-1$, $x_T\in P_{n+1}$,

$$\begin{array}{c}\hfill p(\omega ,u)=\prod _{\tau =0}^{t-1}p\left(x_{\tau +1}\right|x_{\tau },u)·\prod _{\tau =t}^{T-1}p\left(x_{\tau +1}\right|x_{\tau },u)=\\ \hfill =p({{\displaystyle \underline{\omega }}}^{x_t},u)·p({\omega }^{x_t},u)=p({{\displaystyle \underline{\omega }}}^{x_t},u^D)·p({\omega }^{x_t},u^{x_t}),\end{array}$$

(4)

where ${{\displaystyle \underline{\omega }}}^{x_t}=(x_0,x_1,\dots ,x_{t-1},x_t)$ denotes a fragment of trajectory $\omega$ implemented before the subgame ${\mathrm{\Gamma }}^{x_t}$ starts, and $p({{\displaystyle \underline{\omega }}}^{x_t},u)=p(x_t,u)$ denotes the probability that node $x_t$ is reached when the players employ the strategies $u_i$, $i\in N$. It is worth noting that factor-game ${\mathrm{\Gamma }}^D={\mathrm{\Gamma }}^D\left(u^{x_t}\right)$ is usually defined for given strategy profile $u^{x_t}$ in the subgame ${\mathrm{\Gamma }}^{x_t}$ since we assume that

$$h_i^D(x_0,x_1,\dots ,x_{t-1},x_t)=\sum _{\tau =0}^{t-1}{h}_i\left(x_{\tau }\right)+H_i^{x_t}\left(u^{x_t}\right)={\tilde{h}}_i({{\displaystyle \underline{\omega }}}^{x_t}\backslash \left\{x_t\right\})+H_i^{x_t}\left(u^{x_t}\right)$$

(5)

(see, e.g., References [17,22] for details). Moreover, given intermediate node $x_t$, the bundle $\mathrm{\Omega }\left(u\right)=\left\{{\omega }_k\left(u\right)\right|p({\omega }_k,u)>0\}$ can be divided in two subsets, that is, $\mathrm{\Omega }\left(u\right)=\left\{{\mathrm{\Psi }}_m\right\}\cup \left\{{\chi }_l\right\},$ where $x_t\in {\mathrm{\Psi }}_m$, and $x_t\notin {\chi }_l$, $\left\{{\mathrm{\Psi }}_m\right\}\cap \left\{{\chi }_l\right\}=\varnothing$. Then, taking (1), (3), (4) and (5) into account, we get

$$\begin{array}{c}{H}_i\left(u\right)={\displaystyle \sum _m}p({\mathrm{\Psi }}_m,u)·{\tilde{h}}_i\left({\mathrm{\Psi }}_m\right)+{\displaystyle \sum _l}p({\chi }_l,u)·{\tilde{h}}_i\left({\chi }_l\right)=\hfill \\ ={\displaystyle \sum _m}p(x_t,u)·p({\mathrm{\Psi }}_m^{x_t},u^{x_t})·\left[{\tilde{h}}_i({{\displaystyle \underline{\mathrm{\Psi }}}}_m^{x_t}\backslash \left\{x_t\right\})+{\tilde{h}}_i\left({\mathrm{\Psi }}_m^{x_t}\right)\right]+\hfill \\ +{\displaystyle \sum _l}p({\chi }_l,u)·{\tilde{h}}_i\left({\chi }_l\right)=p(x_t,u^D)·{\tilde{h}}_i(x_0,\dots ,x_{t-1})·{\displaystyle \sum _m}p({\mathrm{\Psi }}_m^{x_t},u^{x_t})+\hfill \\ +p(x_t,u^D)·{\displaystyle \sum _m}p({\mathrm{\Psi }}_m^{x_t},u^{x_t})·{\tilde{h}}_i\left({\mathrm{\Psi }}_m^{x_t}\right)+{\displaystyle \sum _l}p({\chi }_l,u)·{\tilde{h}}_i\left({\chi }_l\right)=\hfill \\ =p(x_t,u^D)·{\tilde{h}}_i(x_0,\dots ,x_{t-1})+p(x_t,u^D)·H_i^{x_t}\left(u^{x_t}\right)+\hfill \\ +{\displaystyle \sum _l}p({\chi }_l,u)·{\tilde{h}}_i\left({\chi }_l\right)=p(x_t,u^D)·h_i^D(x_0,\dots ,x_t)+{\displaystyle \sum _l}p({\chi }_l,u)·{\tilde{h}}_i\left({\chi }_l\right).\hfill \end{array}$$

(6)

Note that, since $P_i=P_i^{x_t}\cup P_i^D$, one can compose the ith player’s pure strategy $W_i=(u_i^D,v_i^{x_t})\in U_i$ in the original game ${\mathrm{\Gamma }}^{x_0}$ from her strategies $v_i^{x_t}\in U_i^{x_t}$ in the subgame ${\mathrm{\Gamma }}^{x_t}$ and $u_i^D\in U_i^D$ in the factor-game ${\mathrm{\Gamma }}^D$ [17,22].

3. Refined Backwards Induction Procedure to Construct a Unique SPE

Definition 1

([45]). A strategy profile $u=(u_1,u_2,\dots ,u_n)$ is a Nash Equilibrium (NE) in ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$, if

$$H_i(v_i,u_{-i})⩽H_i(u_i,u_{-i}),\forall v_i\in U_i,\forall i\in N.$$

Let $NE\left({\mathrm{\Gamma }}^{x_0}\right)$ denote the set of all pure strategy Nash equilibria in ${\mathrm{\Gamma }}^{x_0}$.

Definition 2

([25]). A strategy profile u is a subgame perfect (Nash) equilibrium (SPE) in ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$, if $\forall x\in P\backslash P_{n+1}$ it holds that $u^x\in NE\left({\mathrm{\Gamma }}^x\right)$, i. e. the restriction of u on each subgame ${\mathrm{\Gamma }}^x$ forms a $NE$ in this subgame.

To construct SPE in an extensive-form game with perfect information one may employ a so-called backwards induction procedure (see, e.g., References [12,17,22,23,46,47]).

However, the backwards induction procedure may generate multiple subgame perfect equilibriums in an extensive form game with different payoffs to the players (see, e.g., References [5,12,17,23]). To choose a unique SPE and unique corresponding bundle of trajectories we use an approach based on the players’ attitude vectors. Namely, let the ith player’s attitude vector $F_i=\{f_i\left(1\right),\dots ,f_i\left(n\right)\}$ be a permutation of numbers $\{1,\dots ,n\}$ meeting the condition $f_i\left(i\right)=1$. If $f_i\left(j\right)=k$ one may interpret the player j as an "ith player’s associate of level k".

In the paper we will use these attitude vectors when constructing SPE via backwards induction procedure in the following way. Let $x\in P_i$, $H_i^y\left({{\displaystyle \underline{u}}}^y\right)$ denote the ith player’s expected payoff in the subgame ${\mathrm{\Gamma }}^y$, $y\in S\left(x\right)$ while ${{\displaystyle \underline{u}}}^y$ be a SPE in this subgame. Assume that there exist multiple nodes $y_1,\dots ,y_q$ such that $h_i\left(y_1\right)+H_i^{y_1}\left({{\displaystyle \underline{u}}}^{y_1}\right)=\dots =h_i\left(y_q\right)+H_i^{y_q}\left({{\displaystyle \underline{u}}}^{y_q}\right)$, that is, player i is indifferent to the choice of particular node $\overline{y}$ from $\{y_1,\dots ,y_q\}$ while the ith player’s choice may affect the other players’ payoffs. If $f_i\left(j\right)=2$, suppose that the ith player aims to maximize firstly the jth player’s expected payoff $H_j^y\left({{\displaystyle \underline{u}}}^y\right)$ when choosing a unique node y from $y_1,\dots ,y_q$. If again there are several nodes y with the same value $H_j^y\left({{\displaystyle \underline{u}}}^y\right)$ the ith player purposes to maximize secondarily the expected payoff $H_l^y\left({{\displaystyle \underline{u}}}^y\right)$ of such player l that $f_i\left(l\right)=3$, and so on. Note that similar approach to construct a unique SPE in extensive-form game with perfect information but without chance moves was explored in References [17,26,27] for the case when the payoffs are only determined in terminal nodes.

Now let us provide a rigorous specification of this backwards induction procedure refinement which we will refer to as the Attitude SPE or A-SPE algorithm.

Attitude SPE algorithm. Suppose that the players attitude vectors $F_1,F_2,\dots ,F_n$ are of common knowledge, i. e. each player knows these vectors, and all the players are aware of it. Let the length of the trajectory $\omega =(x_0,\dots ,x_t,x_{t+1},\dots ,x_T)$ equals to $T-1$, and the multistage game ${\mathrm{\Gamma }}^{x_0}$ length equals to the maximal length of the trajectory $\omega$ in ${\mathrm{\Gamma }}^{x_0}$. We’ll construct the unique subgame perfect equilibrium ${\displaystyle \underline{u}}=({{\displaystyle \underline{u}}}_1,\dots ,{{\displaystyle \underline{u}}}_n)$ in ${\mathrm{\Gamma }}^{x_0}$ by induction in the length L of the subgame ${\mathrm{\Gamma }}^x$.

Step 

$\mathit{L}=\mathbf{1}$: Consider a subgame ${\mathrm{\Gamma }}^x$ of the length $L=1$. If $x\in P_i$, $i=1,\dots ,n$, we have two cases.

Case 1:

there exists a unique $z^k\in S\left(x\right)=P_{n+1}^x$ such that $h_i\left(z^k\right)={\displaystyle \underset{z\in S\left(x\right)}{max}}{h}_i\left(z\right).$ Then suppose that ${{\displaystyle \underline{u}}}_i\left(x\right)=z^k,p\left(z^k\right|x,{\displaystyle \underline{u}})=1,p\left(z\right|x,{\displaystyle \underline{u}})=0\forall z\in S\left(x\right)\backslash \left\{z^k\right\}.$

Case 2:

there exist $q>1$ nodes $z^{k_q}\in S\left(x\right)=P_{n+1}^x$ such that $h_i\left(z^{k_1}\right)=h_i\left(z^{k_2}\right)=\dots =h_i\left(z^{k_q}\right)={\displaystyle \underset{z\in S\left(x\right)}{max}}{h}_i\left(z\right).$ Then suppose that the ith player chooses the terminal position $z^k\in \{z^{k_1},\dots ,z^{k_q}\}=S^{i,1}\left(x\right)$ such that

$$h_j\left(z^k\right)={\displaystyle \underset{z\in S^{i,1}\left(x\right)}{max}}{h}_j\left(z\right),\mathrm{where}{f}_i\left(j\right)=2.$$

(7)

Let $S^{i,2}\left(x\right)$ denote the set of all nodes $z^k\in S^{i,1}\left(x\right)$ meeting (7). If $S^{i,2}\left(x\right)$ consists of unique node $z^k$ then ${{\displaystyle \underline{u}}}_i\left(x\right)=z^k$, $p\left(z^k\right|x,{\displaystyle \underline{u}})=1$, $p\left(z\right|x,{\displaystyle \underline{u}})=0$$\forall z\in S\left(x\right)\backslash \left\{z^k\right\}$. Otherwise, suppose that the ith player chooses terminal node $z^k\in S^{i,2}\left(x\right)$ such that

$$h_l\left(z^k\right)={\displaystyle \underset{z\in S^{i,2}\left(x\right)}{max}}{h}_l\left(z\right),\mathrm{where}{f}_i\left(l\right)=3.$$

(8)

Let $S^{i,3}\left(x\right)$ denote the set of all final nodes $z^k\in S^{i,2}\left(x\right)$ satisfying (8), and so on.

⋮

Finally, if $S^{i,n}\left(x\right)$ contains unique node $z^k$, then ${{\displaystyle \underline{u}}}_i\left(x\right)=z^k$, $p\left(z^k\right|x,{\displaystyle \underline{u}})=1$, $p\left(z\right|x,{\displaystyle \underline{u}})=0$$\forall z\in S\left(x\right)\backslash \left\{z^k\right\}$. Otherwise, suppose that player i chooses the final node $z^k$ from $S^{i,n}\left(x\right)$ with minimal ordinal number k.

Note that for all cases $H_j\left({\displaystyle \underline{u}}\right)=h_j\left(z^k\right)$, $j\in N$.

If $x\in P_0$ then $S\left(x\right)=P_{n+1}^x$ and we do not need to define a strategy of any player at x, while $H_j\left({\displaystyle \underline{u}}\right)={\displaystyle \sum _{z^k\in S\left(x\right)}}\pi \left(z^k\right|x)·h_j\left(z^k\right).$ Hence, the players’ behavior ${{\displaystyle \underline{u}}}^x=({{\displaystyle \underline{u}}}_1^x,\dots ,{{\displaystyle \underline{u}}}_n^x)\in NE\left({\mathrm{\Gamma }}^x\right)$ and the expected payoffs $H_j^x\left({{\displaystyle \underline{u}}}^x\right),j\in N$ are defined for all subgames ${\mathrm{\Gamma }}^x$ of the length 1. In addition, for games ${\mathrm{\Gamma }}^y$, $y\in P_{n+1}$ of length $L=0$ we assume that $H_i^y\left({{\displaystyle \underline{u}}}^y\right)=h_i\left(y\right)$, $i\in N$.

Step 

$\mathbf{2},\dots ,\mathit{L}-\mathbf{1}$: Suppose that at each subgame ${\mathrm{\Gamma }}^y$ of the length $(L-1)$ or less the unique SPE ${{\displaystyle \underline{u}}}^y=({{\displaystyle \underline{u}}}_1^y,\dots ,{{\displaystyle \underline{u}}}_n^y)$ has been already constructed (“inductive assumption”), and $H_i^y\left({{\displaystyle \underline{u}}}^y\right)$, $i\in N$, is the corresponding vector of all the players’ payoffs.

Step 

$\mathit{L}$: Consider the game ${\mathrm{\Gamma }}^{x_0}$ of the length $L⩾1$. Note that for all $y\in S\left(x_0\right)$ the length of the subgame ${\mathrm{\Gamma }}^y$ is less than L. If $x_0\in P_0$ then

$$H_j\left({\displaystyle \underline{u}}\right)={\displaystyle \sum _{y\in S\left(x_0\right)}}\pi \left(y\right|x_0)·(h_j\left(y\right)+H_j^y\left({{\displaystyle \underline{u}}}^y\right))⩾{\displaystyle \sum _{y\in S\left(x_0\right)}}\pi \left(y\right|x_0)·(h_j\left(y\right)+H_j^y(u_j^y,{{\displaystyle \underline{u}}}_{-j}^y))=H_j(u_j^y,{{\displaystyle \underline{u}}}_{-j}^y)$$

(9)

for all $u_j=u_j^y\in U_j^y=U_j$ since ${{\displaystyle \underline{u}}}^y\in NE\left({\mathrm{\Gamma }}^y\right)$ due to induction assumption, and each player $j\in N$ can deviate from ${{\displaystyle \underline{u}}}_j$ only in the subgames ${\mathrm{\Gamma }}^y$, $y\in S\left(x_0\right)$.

If $x_0\in P_i$ for some $i\in N$, we have two cases.

Case 1:

there exists a unique $\overline{y}\in S\left(x_0\right)$ such that

$$h_i\left(\overline{y}\right)+H_i^{\overline{y}}\left({{\displaystyle \underline{u}}}^{\overline{y}}\right)={\displaystyle \underset{y\in S\left(x_0\right)}{max}}\left(h_i\left(y\right)+H_i^y\left({{\displaystyle \underline{u}}}^y\right)\right).$$

(10)

Then we suppose that ${{\displaystyle \underline{u}}}_i\left(x_0\right)=\overline{y};{{\displaystyle \underline{u}}}_j\left(x\right)={{\displaystyle \underline{u}}}_j^y\left(x\right)$ if $x\in P_j\cap K^y,y\in S\left(x_0\right)$, $j=1,\dots ,n$.

Case 2:

there exist $q>1$ nodes ${\overline{y}}_1,\dots ,{\overline{y}}_q\in S\left(x_0\right)$ such that

$$h_i\left({\overline{y}}_1\right)+H_i^{{\overline{y}}_1}\left({{\displaystyle \underline{u}}}^{{\overline{y}}_1}\right)=\dots =h_i\left({\overline{y}}_q\right)+H_i^{{\overline{y}}_q}\left({{\displaystyle \underline{u}}}^{{\overline{y}}_q}\right)={\displaystyle \underset{y\in S\left(x_0\right)}{max}}\left(h_i\left(y\right)+H_i^y\left({{\displaystyle \underline{u}}}^y\right)\right).$$

(11)

Then we suppose that the ith player chooses $\overline{y}\in \{{\overline{y}}_1,\dots ,{\overline{y}}_q\}=S^{i,1}\left(x_0\right)$ such that

$$h_j\left(\overline{y}\right)+H_j^{\overline{y}}\left({\overline{u}}^{\overline{y}}\right)={\displaystyle \underset{y\in S^{i,1}\left(x_0\right)}{max}}\left(h_j\left(y\right)+H_j^y\left({{\displaystyle \underline{u}}}^y\right)\right),\mathrm{where}{f}_i\left(j\right)=2.$$

(12)

Let $S^{i,2}\left(x_0\right)$ denote the set of all nodes $\overline{y}\in S^{i,1}\left(x_0\right)$ satisfying (12). If $S^{i,2}\left(x_0\right)$ consists of unique node $\overline{y}$ then we suppose that ${{\displaystyle \underline{u}}}_i\left(x_0\right)=\overline{y}$; ${{\displaystyle \underline{u}}}_j\left(x\right)={{\displaystyle \underline{u}}}_j^y\left(x\right)$ if $x\in P_j\cap K^y$, $y\in S\left(x_0\right)$, $j=1,\dots ,n$. Otherwise, suppose that the ith player chooses node $\overline{y}\in S^{i,2}\left(x_0\right)$ such that

$$h_l\left(\overline{y}\right)+H_l^{\overline{y}}\left({\overline{u}}^{\overline{y}}\right)={\displaystyle \underset{y\in S^{i,2}\left(x_0\right)}{max}}\left(h_l\left(y\right)+H_l^y\left({{\displaystyle \underline{u}}}^y\right)\right),\mathrm{where}{f}_i\left(l\right)=3.$$

(13)

Let $S^{i,3}\left(x_0\right)$ denote the set of all nodes $\overline{y}\in S^{i,2}\left(x_0\right)$ meeting (13), and so on.

⋮

Finally, if $S^{i,n}\left(x_0\right)$ contains several nodes ${\overline{y}}_m$, denote by ${\displaystyle \underline{l}}={\displaystyle \underset{{\overline{y}}_m\in S^{i,n}\left(x_0\right)}{min}}\left\{l\right|z^l\in P_{n+1}^{{\overline{y}}_m}\cap \mathrm{\Omega }\left({{\displaystyle \underline{u}}}^{{\overline{y}}_m}\right)\}$ the minimal number of terminal nodes of the trajectories generated by subgame perfect equilibriums ${{\displaystyle \underline{u}}}^{{\overline{y}}_m}$ in the subgames ${\mathrm{\Gamma }}^{{\overline{y}}_m}$, ${\overline{y}}_m\in S^{i,n}\left(x_0\right)$ (see Remark 1). Note that there exists unique trajectory $\omega =(x_0,\dots ,z^{{\displaystyle \underline{l}}})$ from $x_0$ to $z^{{\displaystyle \underline{l}}}$ in the game ${\mathrm{\Gamma }}^{x_0}$, and let $\overline{y}=\omega \cap S^{i,n}\left(x_0\right)$. Again, we suppose that ${{\displaystyle \underline{u}}}_i\left(x_0\right)=\overline{y}$; ${{\displaystyle \underline{u}}}_j\left(x\right)={{\displaystyle \underline{u}}}_j^y\left(x\right)$ if $x\in P_j\cap K^y$, $y\in S\left(x_0\right)$, $j=1,\dots ,n$.

Now we prove that for both cases no player has profitable deviation in ${\mathrm{\Gamma }}^{x_0}$ from the strategy profile ${\displaystyle \underline{u}}=({{\displaystyle \underline{u}}}_1,\dots ,{{\displaystyle \underline{u}}}_n)$ constructed above.

$$H_i\left({\displaystyle \underline{u}}\right)=h_i\left(\overline{y}\right)+H_i^{\overline{y}}\left({{\displaystyle \underline{u}}}^{\overline{y}}\right)⩾h_i\left(y\right)+H_i^y\left({{\displaystyle \underline{u}}}^y\right)⩾h_i\left(y\right)+H_i^y(u_i^y,{{\displaystyle \underline{u}}}_{-i}^y)$$

(14)

for all $y\in S\left(x_0\right)$, $u_i^y\in U_i^y$ due to (10), (11) and the induction assumption that ${{\displaystyle \underline{u}}}^y\in NE\left({\mathrm{\Gamma }}^y\right)$, $y\in S\left(x_0\right)$.

For other players $j\in N$, $j\ne i$, we have

$$H_j\left({\displaystyle \underline{u}}\right)=h_j\left(\overline{y}\right)+H_j^{\overline{y}}\left({{\displaystyle \underline{u}}}^{\overline{y}}\right)⩾h_j\left(\overline{y}\right)+H_j^{\overline{y}}(u_j^{\overline{y}},{{\displaystyle \underline{u}}}_{-j}^{\overline{y}})=H_j(u_j,{{\displaystyle \underline{u}}}_{-j})$$

(15)

for all $u_j\in U_j$ since ${{\displaystyle \underline{u}}}^{\overline{y}}\in NE\left({\mathrm{\Gamma }}^{\overline{y}}\right)$, and the only deviation of player $j\in N$, $j\ne i$ from ${{\displaystyle \underline{u}}}_j$ in the subgame ${\mathrm{\Gamma }}^{\overline{y}}$ may affect the players’ payoffs.

Hence, taking (9), (14) and (15) into account we obtain by induction that the strategy profile ${\displaystyle \underline{u}}=({{\displaystyle \underline{u}}}_1,\dots ,{{\displaystyle \underline{u}}}_n)$ constructed above forms unique subgame perfect equilibria in ${\mathrm{\Gamma }}^{x_0}$.

Proposition 1.

If the players attitude vectors $F_1,F_2,...,F_n$ are of common knowledge, the Attitude SPE algorithm allows to construct a unique subgame perfect equilibrium ${\displaystyle \underline{u}}=({{\displaystyle \underline{u}}}_1,\dots ,{{\displaystyle \underline{u}}}_n)$ in pure strategies for any extensive-form game ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$ with chance moves as well as a unique bundle of trajectories $\mathrm{\Omega }\left({\displaystyle \underline{u}}\right)$.

It is worth noting than the existence of (subgame perfect) pure strategy equilibrium in extensive form game with perfect information and chance moves was first proved in References [46,47] for the partial case when the payoffs are only defined in terminal nodes. Hence, Proposition 1 could be considered as a corollary of these results. However, we provide a rigorous algorithm how to construct a unique $SPE$ in extensive-form game with chance moves as well as a (unique) corresponding bundle of trajectories. We will use this algorithm, in particular, to calculate the characteristic function of the cooperative extensive-form game in Section 4.

Let us use the following example to demonstrate how the Attitude SPE algorithm works.

 Example 1.

(A 3-player multistage game with chance moves).

Let $P_0=\{{\overline{x}}_1,{\overline{x}}_3\}$, $P_1=\{{\overline{x}}_0,{\overline{x}}_4^2\}$, $P_2=\{{\overline{x}}_2^1,{\overline{x}}_5\}$, $P_3=\{x_2^2,x_4^3\}$, $P_{n+1}=\{z_1,\dots ,z_{10}\}$. The players’ payoffs and probabilities $\pi \left(y\right|x)$, $x\in P_0$ are written in the game tree.

Suppose that the players’ attitude vectors are $F_1=\left(f_1\left(1\right),f_1\left(2\right),f_1\left(3\right)\right)=(1,3,2)$, $F_2=(2,3,1)$ and $F_3=(3,1,2)$.

When using the Attitude SPE algorithm, at each node $x\in P_i$, $i=1,2,3$, the ith player has to choose the alternative marked in bold violet in Figure 1. Note that

$$H^{x_3}\left({{\displaystyle \underline{u}}}^{x_3}\right)=\left(\begin{array}{c}12\\ 0\\ 0\end{array}\right)+\frac{1}{6}\left(\begin{array}{c}0\\ 24\\ 0\end{array}\right)+\frac{1}{2}\left(\begin{array}{c}24\\ 0\\ 24\end{array}\right)+\frac{1}{3}\left(\begin{array}{c}0\\ 18\\ 12\end{array}\right)=\left(\begin{array}{c}24\\ 10\\ 16\end{array}\right)and{H}^{z_2}=h\left(z_2\right)=\left(\begin{array}{c}0\\ 10\\ 20\end{array}\right).$$

Hence, $S^{2,1}\left(x_2^1\right)=\{z_2,x_3\}$, and ${{\displaystyle \underline{u}}}_2\left(x_2^1\right)=z_2$ due to the player’s 2 attitude vector $F_2$.

The A-SPE algorithm generates unique SPE ${\displaystyle \underline{u}}=({{\displaystyle \underline{u}}}_1,{{\displaystyle \underline{u}}}_2,{{\displaystyle \underline{u}}}_3)$, where ${{\displaystyle \underline{u}}}_1\left(x_0\right)=x_1$, ${{\displaystyle \underline{u}}}_1\left(x_4^2\right)=z_8$; ${{\displaystyle \underline{u}}}_2\left(x_2^1\right)=z_2$, ${{\displaystyle \underline{u}}}_2\left(x_5\right)=z_9$; ${{\displaystyle \underline{u}}}_3\left(x_2^2\right)=z_4$, ${{\displaystyle \underline{u}}}_3\left(x_4^3\right)=z_6$, while $H\left({\displaystyle \underline{u}}\right)=(11,22,18)$. We will use this SPE later in Section 4 when calculating the γ-characteristic function.

4. Cooperative Strategies and Trajectories

If the players agree to cooperate in multicriteria game ${\mathrm{\Gamma }}^{x_0}$, first they are expected to maximize the total payoff ${\displaystyle \sum _{i=1}^n}{H}_i\left(u\right)$ of the grand coalition. Let $\overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ denote the set of all pure strategy profiles u, such that

$${\displaystyle \sum _{i\in N}}{H}_i\left(u\right)={\displaystyle \underset{v\in U}{max}}{\displaystyle \sum _{i\in N}}{H}_i\left(v\right)=\overline{H}.$$

(16)

The set $\overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ is known to be nonempty and it may contain multiple strategy profiles (see, e.g., Reference [17]). Hence, the players need to agree on a specific approach they are going to use to choose a unique optimal cooperative strategy profile $\overline{u}\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ as well as the corresponding optimal bundle of cooperative trajectories in the game tree. To this aim we introduce the so-calle Players’ Rank Based (PRB) algorithm. Note that rather close approach—using the ranking of the criteria to choose a unique cooperative trajectory—was proposed recently in Reference [8] for multicriteria extensive-form games without chance moves. Namely, suppose that the players have agreed on the so-called "rank" of each player within the grand coalition N, and $r\left(k\right)=i$ means that the rank of player i equals k, $k=1,...,n$.

Players’ rank based (PRB) algorithm.

 Step

0. Consider the set $\overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$. If all strategy profiles $u\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ generate the same bundle of trajectories $\mathrm{\Omega }\left(u\right)$ (see, e.g., References [17,22,23] for discussion on a certain redundancy of the pure strategy definition in extensive game), let the players choose any strategy profile $\overline{u}\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ as the cooperative strategy profile and $\mathrm{\Omega }\left(\overline{u}\right)$ denote the corresponding bundle of cooperative trajectories.

 Step

$\mathit{k}=\mathbf{1}$. Otherwise, that is, if the strategy profiles from $\overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ generate different (and hence, disjoint—see Remark 1) bundles of the trajectories, calculate

$${\displaystyle \underset{v\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)}{max}}{H}_{r\left(1\right)}\left(v\right)={\overline{H}}_{r\left(1\right)}.$$

Let ${\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$ denote the set of all strategy profiles u such that $H_{r\left(1\right)}\left(u\right)={\overline{H}}_{r\left(1\right)}$. If all strategy profiles $u\in {\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$ generate the same bundle of trajectories $\mathrm{\Omega }\left(u\right)$, the players may choose any strategy profile $\overline{u}\in {\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$ as the cooperative strategy profile. Otherwise proceed to the next step.

 Step

$\mathit{k}=\mathbf{2}$. Consider the set ${\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$. If all strategy profiles $u\in {\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$ generate the same bundle of trajectories $\mathrm{\Omega }\left(u\right)$, the players may choose any strategy profile $\overline{u}\in {\overline{U}}_{r\left(1\right)}\left({\mathrm{\Gamma }}^{x_0}\right)$ as the cooperative strategy profile. Otherwise, proceed to the next step.

 Step

$\mathit{k}$ ($\mathit{k}=\mathbf{2},\dots ,\mathit{n}$).

 ⋮

 

 Step

$\mathit{k}+\mathbf{1}$. Finally, if the strategy profiles from $u\in {\overline{U}}_{r\left(n\right)\left({\mathrm{\Gamma }}^{x_0}\right)}$ generate different bundles of the trajectories, we suppose that the players choose such $\overline{u}\in {\overline{U}}_{r\left(n\right)\left({\mathrm{\Gamma }}^{x_0}\right)}$ that $\mathrm{\Omega }\left(\overline{u}\right)=\{{\omega }_m\left(\overline{u}\right)=(x_0,\dots ,x_{T\left(m\right)}=z_l)|p({\omega }_m,\overline{u})>0\}$ contains the trajectory $\omega \left(\overline{u}\right)$ with minimal number l of the terminal node $z_l$ (see Remark 1).

Henceforth, we will refer to the strategy profile $\overline{u}\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ and the bundle of the trajectories $\mathrm{\Omega }\left(\overline{u}\right)$ as the optimal cooperative strategy profile and the optimal bundle of cooperative trajectories respectively.

In the dynamic setting it is significant that a specific method which the players agreed to accept in order to choose a unique optimal cooperative strategy profile $\overline{u}\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$ as well as the corresponding optimal bundle of cooperative trajectories satisfies time consistency (see, e.g., References [1,2,6,13,17]), that is, a fragment of the optimal bundle of the cooperative trajectories in the subgame should remain optimal in this subgame. Suppose that at each subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$ along the cooperative trajectories, that is ${\overline{x}}_t\in \omega \left(\overline{u}\right)$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, the players choose the strategy profile $u^{{\overline{x}}_t}\in U^{{\overline{x}}_t}$ such that

$$u^{{\overline{x}}_t}\in {\displaystyle \underset{v^{{\overline{x}}_t}\in U^{{\overline{x}}_t}}{arg\; max}}{\displaystyle \sum _{i\in N}}{H}_i^{{\overline{x}}_t}\left(v^{{\overline{x}}_t}\right).$$

(17)

Let $\overline{U}\left({\mathrm{\Gamma }}^{{\overline{x}}_t}\right)$ denote the set of all pure strategy profiles $u^{{\overline{x}}_t}\in U^{{\overline{x}}_t}$ which satisfy (17) and the players use the same approach to choose a unique optimal cooperative strategy profile ${\overline{u}}^{{\overline{x}}_t}\in \overline{U}\left({\mathrm{\Gamma }}^{{\overline{x}}_t}\right)$ in the subgame as for the original game ${\mathrm{\Gamma }}^{x_0}$ (namely, the PRB algorithm).

Proposition 2.

A cooperative strategy profile for ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$ based on the PRB algorithm satisfies time consistency. Namely, let $\overline{u}\in U$ satisfies (16), and $\mathrm{\Omega }\left(\overline{u}\right)$ be the optimal bundle of cooperative trajectories. Then for each subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$, ${\overline{x}}_t\in \omega \left(\overline{u}\right)=({\overline{x}}_0,\dots ,{\overline{x}}_t,{\overline{x}}_{t+1},\dots ,{\overline{x}}_T),1⩽t<T,$ with ${\overline{x}}_0=x_0$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, it holds that

$${\displaystyle \sum _{i\in N}}{H}_i^{{\overline{x}}_t}\left({\overline{u}}^{{\overline{x}}_t}\right)={\displaystyle \underset{v^{{\overline{x}}_t}\in U^{{\overline{x}}_t}}{max}}{\displaystyle \sum _{i\in N}}{H}_i^{{\overline{x}}_t}\left(v^{{\overline{x}}_t}\right),$$

(18)

while ${\omega }^{{\overline{x}}_t}=({\overline{x}}_t,{\overline{x}}_{t+1},\dots ,{\overline{x}}_T)\in \mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_t}\right)$, that is, ${\omega }^{{\overline{x}}_t}$ belongs to the optimal bundle of cooperative trajectories in the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$.

Proof. 

The optimal bundle of cooperative trajectories $\mathrm{\Omega }\left(\overline{u}\right)$ generated by $\overline{u}\in \overline{PO}\left({\mathrm{\Gamma }}^{x_0}\right)$ can be divided onto two subsets $\left\{{\mathrm{\Psi }}_m\right\}=\{\omega \in \mathrm{\Omega }\left(\overline{u}\right)|{\overline{x}}_t\in \omega \}$ and $\left\{{\chi }_l\right\}=\{\omega \in \mathrm{\Omega }\left(\overline{u}\right)|{\overline{x}}_t\notin \omega \}$ while $\left\{{\mathrm{\Psi }}_m\right\}\cap \left\{{\chi }_l\right\}=\varnothing$, $\left\{{\mathrm{\Psi }}_m\right\}\cup \left\{{\chi }_l\right\}=\mathrm{\Omega }\left(\overline{u}\right)$. Then, taking (5) and (6) into account we get

$$\begin{array}{cc}\hfill & H_i\left(\overline{u}\right)={\displaystyle \sum _m}p({\mathrm{\Psi }}_m,\overline{u})·{\tilde{h}}_i\left({\mathrm{\Psi }}_m\right)+{\displaystyle \sum _l}p({\chi }_l,\overline{u})·{\tilde{h}}_i\left({\chi }_l\right)=\hfill \\ \hfill & =p({\overline{x}}_t,\overline{u})·\left[{\tilde{h}}_i({\overline{x}}_0,{\overline{x}}_1,\dots ,{\overline{x}}_{t-1})+H_i^{{\overline{x}}_t}\left({\overline{u}}^{{\overline{x}}_t}\right)\right]+{\displaystyle \sum _l}p({\chi }_l,{\overline{u}}^D)·{\tilde{h}}_i\left({\chi }_l\right),\hfill \end{array}$$

(19)

and (16) for $\overline{u}$ takes the form

$$\begin{array}{c}{\displaystyle \sum _{i\in N}}p({\overline{x}}_t,\overline{u})·\left({\tilde{h}}_i({\overline{x}}_0,{\overline{x}}_1,\dots ,{\overline{x}}_{t-1})+H_i^{{\overline{x}}_t}\left({\overline{u}}^{{\overline{x}}_t}\right)\right)+\hfill \\ +{\displaystyle \sum _{i\in N}}{\displaystyle \sum _l}p({\chi }_l,{\overline{u}}^D)·{\tilde{h}}_i\left({\chi }_l\right)={\displaystyle \underset{v\in U}{max}}{H}_i\left(v\right).\hfill \end{array}$$

(20)

Suppose that ${\overline{u}}^{{\overline{x}}_t}$ does not satisfy (18), that is, there exists $v^{{\overline{x}}_t}\in U^{{\overline{x}}_t}$ such that

$${\displaystyle \sum _{i\in N}}{H}_i^{{\overline{x}}_t}\left({\overline{u}}^{{\overline{x}}_t}\right)<{\displaystyle \sum _{i\in N}}{H}_i^{{\overline{x}}_t}\left(v^{{\overline{x}}_t}\right).$$

(21)

Denote by $\mathrm{\Omega }\left(v^{{\overline{x}}_t}\right)=\{{\lambda }_m^{{\overline{x}}_t}=({\overline{x}}_t,\dots ,{\overline{x}}_{T\left(m\right)})|p({\lambda }_m^{{\overline{x}}_t},v^{{\overline{x}}_t})>0\}$ the bundle of all trajectories in the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$ generated by $v^{{\overline{x}}_t}$. Then (21) takes the form

$${\displaystyle \sum _{i\in N}}{\displaystyle \sum _m}p({\mathrm{\Psi }}_m^{{\overline{x}}_t},{\overline{u}}^{{\overline{x}}_t})·{\tilde{h}}_i^{{\overline{x}}_t}\left({\mathrm{\Psi }}_m^{{\overline{x}}_t}\right)<{\displaystyle \sum _{i\in N}}{\displaystyle \sum _m}p({\lambda }_m^{{\overline{x}}_t},v^{{\overline{x}}_t})·{\tilde{h}}_i^{{\overline{x}}_t}\left({\lambda }_m^{{\overline{x}}_t}\right).$$

(22)

Denote by $W_i=({\overline{u}}_i^D,v_i^{{\overline{x}}_t})$, $i\in N$, the ith player’s compound pure strategy in ${\mathrm{\Gamma }}^{x_0}$. The strategy profile $W=(W_1,\dots ,W_n)$ generates the strategy bundle $\mathrm{\Omega }\left(W\right)$ that can be divided onto two disjoint subsets $\left\{{\lambda }_m\right\}=\{\omega \in \mathrm{\Omega }\left(W\right)|{\overline{x}}_t\in \omega \}$ and $\left\{{\chi }_l\right\}=\{\omega \in \mathrm{\Omega }\left(W\right)|{\overline{x}}_t\notin \omega \}$, where the second subset for $\mathrm{\Omega }\left(W\right)$ coincides with the second subset for $\mathrm{\Omega }\left(\overline{u}\right)$ since $W^D=u^D$, and ${\lambda }_m=({\overline{x}}_0,\dots ,{\overline{x}}_t)\cup ({\overline{x}}_t,\dots ,{\overline{x}}_{T\left(m\right)})=({\overline{x}}_0,\dots ,{\overline{x}}_t)\cup {\lambda }_m^{{\overline{x}}_t}.$

Adding ${\displaystyle \sum _{i\in N}}{\tilde{h}}_i({\overline{x}}_0,{\overline{x}}_1\dots ,{\overline{x}}_{t-1})$ to both sides of (22) we get

$${\displaystyle \sum _{i\in N}}\left({\tilde{h}}_i({\overline{x}}_0,\dots ,{\overline{x}}_{t-1})+H_i^{{\overline{x}}_t}\left({\overline{u}}^{{\overline{x}}_t}\right)\right)<{\displaystyle \sum _{i\in N}}\left({\tilde{h}}_i({\overline{x}}_0,\dots ,{\overline{x}}_{t-1})+H_i^{{\overline{x}}_t}\left(v^{{\overline{x}}_t}\right)\right).$$

(23)

Then we can multiply both sides of (23) on $p({\overline{x}}_t,\overline{u})=p({\overline{x}}_t,{\overline{u}}^D)=p({\overline{x}}_t,W^D)=p({\overline{x}}_t,W)>0$ and then add ${\displaystyle \sum _{i\in N}}{\displaystyle \sum _l}p({\chi }_l,{\overline{u}}^D)·{\tilde{h}}_i\left({\chi }_l\right)$ to both sides of the last inequality. Taking into account (4)–(6) and (20) we obtain

$${\displaystyle \sum _{i\in N}}{H}_i\left(\overline{u}\right)<{\displaystyle \sum _{i\in N}}{H}_i\left(W\right)$$

for the constructed strategy profile $W\in U$. The last inequality contradicts the fact that $\overline{u}\in \overline{U}\left({\mathrm{\Gamma }}^{x_0}\right)$, hence (18) is valid.

Arguing in a similar way (for the case when different strategy profiles from $\overline{U}\left({\mathrm{\Gamma }}^{{\overline{x}}_t}\right)$ generate different bundles of the trajectories) we can verify that ${\omega }^{{\overline{x}}_t}=({\overline{x}}_t,\dots ,{\overline{x}}_T)$ — a fragment of the cooperative trajectory $\omega \in \mathrm{\Omega }\left(\overline{u}\right)$, starting at ${\overline{x}}_t$ — belongs to the optimal bundle of cooperative trajectories in the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$, that is, ${\omega }^{{\overline{x}}_t}\in \mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_t}\right)$. □

We will assume in this paper that all the players have agreed to apply the PRB algorithm in order to choose the cooperative strategy profile $\overline{u}=({\overline{u}}_1,\dots ,{\overline{u}}_n)$ that generates the optimal bundle $\mathrm{\Omega }\left(\overline{u}\right)$ of cooperative trajectories in ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$. The next step of cooperation is to define a characteristic function $V^{x_0}\left(S\right)$. There are different notions of characteristic functions (see, e.g., References [23,24,48]), in this paper we adopt the so-called $\gamma$-characteristic function introduced in Reference [24]. Namely, we assume that $V^{x_0}\left(S\right)$ is given by the SPE (based on the Attitude SPE algorithm) outcome of S in the noncooperative game between members of S maximizing their joint payoff, and non members playing individually.

The $\gamma$-characteristic function $V^{{\overline{x}}_t}$ for the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$, ${\overline{x}}_t\in {\omega }_m\left(\overline{u}\right)=({\overline{x}}_0,\dots ,{\overline{x}}_t,$$\dots ,{\overline{x}}_{T\left(m\right)})$, ${\omega }_m\left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$ along the optimal bundle of cooperative trajectories can be constructed using the same approach. Note that

$$V^{{\overline{x}}_t}\left(N\right)={\displaystyle \sum _{{\omega }_m^{{\overline{x}}_t}\in \mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_t}\right)}}p({\omega }_m^{{\overline{x}}_t},{\overline{u}}^{{\overline{x}}_t})·\sum _{\tau =t}^{T\left(m\right)}{\displaystyle \sum _{i\in N}}{h}_i\left({\overline{x}}_{\tau }\right),t=0,1,\dots ,T\left(m\right).$$

(24)

Let ${\mathrm{\Gamma }}^{x_0}\left(N,V^{x_0}\right)$ denote extensive-form cooperative game ${\mathrm{\Gamma }}^{x_0}\in G^{cm}\left(n\right)$ with $\gamma$-characteristic function, and ${\mathrm{\Gamma }}^{{\overline{x}}_t}\left(N,V^{{\overline{x}}_t}\right)$ denote the corresponding subgame.

We assume that the players adopt a single-valued cooperative solution ${\phi }^{x_0}$ (for instance, the Shapley value [33], the nucleolus [34], etc.) for the cooperative game ${\mathrm{\Gamma }}^{x_0}(N,V^{x_0})$ which satisfies the collective rationality property

$${\displaystyle \sum _{i=1}^n{\phi }_i^{x_0}=V^{x_0}\left(N\right)={\displaystyle \sum _{{\omega }_m\in \mathrm{\Omega }\left(\overline{u}\right)}}p({\omega }_m,\overline{u})·\sum _{\tau =0}^{T\left(m\right)}{\displaystyle \sum _{i\in N}}{h}_i\left({\overline{x}}_{\tau }\right)}$$

(25)

and the individual rationality property

$${\phi }_i^{x_0}\geqq V^{x_0}\left(\left\{i\right\}\right),i=1,\dots ,n.$$

(26)

In addition, we assume that the same properties (25) and (26) are valid for the cooperative solutions ${\phi }^{{\overline{x}}_t}$ at each subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}(N,V^{{\overline{x}}_t})$, $t=0,\dots ,T-1$.

It is worth noting that the last assumption as well as the choice of $\gamma$-characteristic function ensure that every player has an incentive to cooperate at each subgame along the optimal game evolution since the ith player’s cooperative payoff-to-go at ${\mathrm{\Gamma }}^{{\overline{x}}_t}(N,V^{{\overline{x}}_t})$, $t=0,\dots ,T-1$, is at least equal to her non-cooperative counterpart: ${\phi }_i^{{\overline{x}}_t}⩾H_i^{{\overline{x}}_t}\left({{\displaystyle \underline{u}}}^{{\overline{x}}_t}\right)$.

5. Subgame Consistency and Incremental IDP

Let $\beta =\left\{{\beta }_i\left({\overline{x}}_{\tau }\right)\right\}$, $i=1,\dots ,n$; $\tau =1,\dots ,T\left(l\right)$, $\overline{x}\left(\tau \right)\in {\omega }_l\left(\overline{u}\right)$, ${\omega }_l\left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$ denote the Imputation Distribution Procedure (IDP) for the cooperative solution ${\left({\phi }_i^{{\overline{x}}_0}\right)}_{i\in N}$ or the payment schedule (see, e.g., References [3,8,9,10,11,12,14,15,16,17,18,20] for details). The IDP approach means that all the players have agreed to allocate the total cooperative payoff $V^{x_0}\left(N\right)$ between the players along the optimal bundle $\mathrm{\Omega }\left(\overline{u}\right)$ of cooperative trajectories ${\omega }_l\left(\overline{u}\right)$ according to some specific rule which is called IDP. Namely, ${\beta }_i\left({\overline{x}}_{\tau }\right)$ denotes the actual current payment which the player i receives at position ${\overline{x}}_{\tau }$ (instead of $h_i\left({\overline{x}}_{\tau }\right)$) if the players employ the IDP $\beta$. Moreover, one can design such an IDP $\beta$ that all the players will be interested in cooperation in any subgame ${\mathrm{\Gamma }}^{{\overline{x}}_{\tau }}$, $\overline{x}\left(\tau \right)\in {\omega }_l\left(\overline{u}\right)$, ${\omega }_l\left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, that is, at any intermediate time instant.

Definition 3.

The IDP $\beta =\left\{{\beta }_i\left({\overline{x}}_{\tau }\right)\right\}$ satisfies subgame efficiency, if at any intermediate node ${\overline{x}}_t\in \omega \left(\overline{u}\right)$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, $0⩽t<T$, it holds that:

$${\displaystyle \sum _{{\omega }_m^{{\overline{x}}_t}\in \mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_t}\right)}}p({\omega }_m^{{\overline{x}}_t},{\overline{u}}^{{\overline{x}}_t})·\sum _{\tau =t}^{T\left(m\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)={\phi }_i^{{\overline{x}}_t},i\in N.$$

(27)

Equation (27) means that the expected sum of the payments to player i along the optimal subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$ evolution equals to what she is entitled to in this subgame. Then the IDP for each player can be reasonably implemented as a rule for step-by-step allocation of the ith player’s current expected optimal payoff. Note that for $t=0$ the subgame efficiency definition coincides with the efficiency at initial node $x_0$ or the efficiency in the whole game ${\mathrm{\Gamma }}^{x_0}$ condition (see References [9,14,16,20]).

Definition 4

([10]). The IDP $\beta =\left\{{\beta }_i\left({\overline{x}}_{\tau }\right)\right\}$ satisfies the strict balance condition if for each node ${\overline{x}}_{\tau }\in {\omega }_m\left(\overline{u}\right)$, ${\omega }_m\left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$$\forall t=0,\dots ,T\left(m\right)$

$${\displaystyle \sum _{i\in N}}{\beta }_i\left({\overline{x}}_{\tau }\right)={\displaystyle \sum _{i\in N}}{h}_i\left({\overline{x}}_{\tau }\right).$$

(28)

Equation (28) ensures the “admissibility” of the IDP, that is, the sum of payments to the players in any node ${\overline{x}}_{\tau }$ is equal to the sum of payoffs that they can collect in this node.

The next advantageous dynamic property of an IDP—the time consistency, introduced in Reference [3]—was extended to dynamic games played over event trees in References [14,16,20] as well as to multicriteria extensive-form cooperative games (with chance moves) in Reference [9].

To write down properly the time consistency condition for some intermediate node ${\overline{x}}_t\in \omega \left(\overline{u}\right)=({\overline{x}}_0,{\overline{x}}_1,\dots ,{\overline{x}}_{t-1},{\overline{x}}_t,{\overline{x}}_{t+1},\dots ,{\overline{x}}_T)$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, $1⩽t<T$, in multistage game ${\mathrm{\Gamma }}^{x_0}$ with chance moves we need to pay attention to all chance nodes on the path $({\overline{x}}_0,\dots ,{\overline{x}}_{t-1})={{\displaystyle \underline{\omega }}}^{x_t}\backslash \left\{{\overline{x}}_t\right\}$.

Namely, let us numerate the chance nodes from $P_0\cap ({{\displaystyle \underline{\omega }}}^{x_t}\backslash \left\{{\overline{x}}_t\right\})$ in order of their occurrence on the path $({\overline{x}}_0,\dots ,{\overline{x}}_{t-1})$, that is, $y_1={\overline{x}}_{t\left(1\right)}$, $y_2={\overline{x}}_{t\left(2\right)},\dots ,y_{\theta }={\overline{x}}_{t\left(\theta \right)}$, $0⩽t\left(1\right)<t\left(2\right)<\dots <t\left(\theta \right)<t$.

Definition 5

([9]). The IDP $\beta =\left\{{\beta }_{i/k}\left({\overline{x}}_{\tau }\right)\right\}$ for the cooperative solution ${\phi }^{x_0}$ is called time consistent in the whole game ${\mathrm{\Gamma }}^{x_0}\left(N,V^{x_0}\right)\in G^{cm}\left(n\right)$ if at any intermediate node ${\overline{x}}_t\in \omega \left(\overline{u}\right)$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, $1⩽t<T$, for all $i\in N$, it holds that

• case $\theta =0$ (no chance nodes on the path $({\overline{x}}_0,\dots ,{\overline{x}}_{t-1})$):

  $$\sum _{\tau =0}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}={\phi }_i^{x_0},$$

  (29)
• case $\theta =1$ (only one chance node $y_1={\overline{x}}_{t\left(1\right)}$ before ${\overline{x}}_t$):

  $$\begin{array}{c}\hfill \sum _{\tau =0}^{t\left(1\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)+p({\overline{x}}_{t\left(1\right)+1},\overline{u})·\left\{\sum _{\tau =t\left(1\right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}\right\}+\\ \hfill +{\displaystyle \sum _{x^k\in S\left({\overline{x}}_{t\left(1\right)}\right)\backslash \left\{{\overline{x}}_{t\left(1\right)+1}\right\}}}p(x^k,\overline{u})·{\phi }_i^{x^k}={\phi }_i^{x_0},\end{array}$$

  (30)
• case $\theta =2$ (two chance nodes $y_1={\overline{x}}_{t\left(1\right)}$, $y_2={\overline{x}}_{t\left(2\right)}$ before ${\overline{x}}_t$):

$$\begin{array}{c}\sum _{\tau =0}^{t\left(1\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)+p({\overline{x}}_{t\left(1\right)+1},\overline{u})·\{\sum _{\tau =t\left(1\right)+1}^{t\left(2\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)+p\left({\overline{x}}_{t\left(2\right)+1}\right|{\overline{x}}_{t\left(2\right)},\overline{u})\times \hfill \\ \times \left[\sum _{\tau =t\left(2\right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}\right]+{\displaystyle \sum _{x^m\in S\left({\overline{x}}_{t\left(2\right)}\right)\backslash \left\{{\overline{x}}_{t\left(2\right)+1}\right\}}}p\left(x^m\right|{\overline{x}}_{t\left(2\right)},\overline{u})·{\phi }_i^{x^m}\}+\hfill \\ +{\displaystyle \sum _{x^k\in S\left({\overline{x}}_{t\left(1\right)}\right)\backslash \left\{{\overline{x}}_{t\left(1\right)+1}\right\}}}p(x^k,\overline{u})·{\phi }_i^{x^k}={\phi }_i^{x_0},\hfill \end{array}$$

(31)

…

Note that for partial case when ${\overline{x}}_t\in S\left({\overline{x}}_{t\left(1\right)}\right)$, that is, if ${\overline{x}}_t$ follows the chance node ${\overline{x}}_{t\left(1\right)}$ Equation (30) takes the simpler form

$$\sum _{\tau =0}^{t\left(1\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\displaystyle \sum _{x^k\in S\left({\overline{x}}_{t\left(1\right)}\right)}}p(x^k,\overline{u})·{\phi }_i^{x^k}={\phi }_i^{x_0}.$$

A similar note is valid for equation (31), and so forth.

Roughly speaking, Definition 5 implies that the payments collected by the ith player (according to the payment schedule $\beta$) before reaching some intermediate node ${\overline{x}}_t$ plus the expected ith player’s component of the Shapley value in the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$ starting at ${\overline{x}}_t$ plus this player’s expected Shapley value components in other subgames along the cooperative trajectories which do not contain ${\overline{x}}_t$ corresponds to what the player i is entitled to in the original game ${\mathrm{\Gamma }}^{x_0}\left(N,V^{x_0}\right)$.

It is worth noting that Definition 5 indeed provides a reasonable consistency requirements which a good payment schedule $\beta$ should satisfy when the player evaluates IDP $\beta$ at the initial node $x_0$, that is, before the game ${\mathrm{\Gamma }}^{x_0}\left(N,V^{x_0}\right)$ starts (and the words “in the whole game” in Definition 5 properly reflect this feature). However, when the player purposes to evaluate IDP $\beta$ in the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$, that is, after reaching some intermediate node ${\overline{x}}_t$ (in case when $\theta ⩾1$) this player will unlikely take into account the expected future payoffs in all the subgames which are unattainable if the node ${\overline{x}}_t$ has been already reached, that is, the last addends in the LHS of (30) and (31). To overcome this problem we suggest the players to use a notion of subgame consistency—a refinement of time consistency that was firstly proposed in Reference [36] for cooperative stochastic differential games and then extend it to stochastic dynamic games in References [37,38]. Let us provide a rigorous definition of the IDP subgame consistency for extensive-form games with chance moves that is applicable in all the subgames along the optimal bundle of cooperative trajectories.

Definition 6.

The IDP $\beta =\left\{{\beta }_i\left({\overline{x}}_{\tau }\right)\right\}$ is called subgame consistent if at any intermediate node ${\overline{x}}_t\in \omega \left(\overline{u}\right)$, $\omega \left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, $1⩽t⩽T$, for all $i\in N$, it holds that

• case $1⩽t⩽t\left(1\right)$ (no chance nodes before the subgame ${\mathrm{\Gamma }}^{{\overline{x}}_t}$ root ${\overline{x}}_t$):

  $$\sum _{\tau =0}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}={\phi }_i^{x_0},$$

  (32)
• case $t\left(1\right)+1<t⩽t\left(2\right)$ (only one chance node $y_1={\overline{x}}_{t\left(1\right)}$ before ${\overline{x}}_t$):

  $$\sum _{\tau =t\left(1\right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}={\phi }_i^{{\overline{x}}_{t\left(1\right)+1}},$$

  (33)
• case $t\left(2\right)+1<t⩽t\left(3\right)$ (two chance nodes before ${\overline{x}}_t$):

  $$\sum _{\tau =t\left(2\right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}={\phi }_i^{{\overline{x}}_{t\left(2\right)+1}},$$

  (34)
• ⋮
• case $t\left(\theta \right)+1<t⩽T$ (no chance nodes after ${\overline{x}}_t$):

  $$\sum _{\tau =t\left(\theta \right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)+{\phi }_i^{{\overline{x}}_t}={\phi }_i^{{\overline{x}}_{t\left(\theta \right)+1}}.$$

  (35)

The subgame consistency definition differs from the “time consistency in the whole game” property (see References [9,14,16,20]) which is based on an a priori assessment of the ith player’s expected optimal payoff (before the game starts). However, when the players make a decision in the subgame after the chance move occurs they need to recalculate the expected optimal payoff since the original optimal bundle of cooperative trajectories shrinks after each chance node. Note that we can not write out the subgame consistency condition for $t=t\left(1\right)+1,t\left(2\right)+1,...,t\left(\theta \right)+1$, that is, for the nodes ${\overline{x}}_t$ that immediately follow the chance nodes.

One can suggest different imputation distribution procedures that may or may not satisfy the useful properties listed above. The review of different IDP for multistage games (without chance moves) as well as the analysis of their properties can be found in References [10,12,15,17]. Below we consider the refinement of the so-called incremental IDP (see, e.g., References [10,14,16,17,20,21]) that was recently introduced for multistage games with chance moves [9].

Definition 7

([9]). The incremental IDP for the cooperative solution ${\phi }^{x_0}$ in multistage game with chance moves ${\mathrm{\Gamma }}^{x_0}$ is defined as follows:

$${\beta }_i\left(x_t\right)={\phi }_i^{x_t}-{\displaystyle \sum _{x_{t+1}^k\in S\left(x_t\right)}}p\left(x_{t+1}^k\right|x_t,\overline{u})·{\phi }_i^{x_{t+1}^k}$$

(36)

for $x_t\in {\omega }_l\left(\overline{u}\right)=(x_0,\dots ,x_t,\dots ,x_{T\left(l\right)})$, ${\omega }_l\left(\overline{u}\right)\in \mathrm{\Omega }\left(\overline{u}\right)$, $t=0,\dots ,T\left(l\right)-1$;

$${\beta }_i\left(x_{T\left(l\right)}\right)={\phi }_i^{x_{T\left(l\right)}}$$

(37)

for $x_{T\left(l\right)}\in \mathrm{\Omega }\left(\overline{u}\right)\cap P_{n+1}$.

Remark 2.

Formulas (36), (37) are similar to the imputation distribution procedures suggested in References [14,16,20] for (single-criterion) stochastic discrete-time dynamic games played over event trees. If $x_t\in P_i$, $i=1,\dots ,n$ Equation (36) takes the simpler form ${\beta }_i\left(x_t\right)={\phi }_i^{x_t}-{\phi }_i^{x_{t+1}}$, where ${\overline{u}}_i\left(x_t\right)=x_{t+1}$, that coincides with the “classical” incremental IDP.

Let us use again 3-person extensive-form game from Example 1 to demonstrate a proposed scheme of cooperation.

 Example 2.

(Cooperative behavior in 3-player game from Ex. 1).

Suppose that the players have agreed on the following ranks: $r\left(1\right)=1$, $r\left(2\right)=2$ and $r\left(3\right)=3$. When implementing the PRB algorithm we get the optimal bundle $\mathrm{\Omega }\left(\overline{u}\right)$ which contains four cooperative trajectories (marked in bold, deep blue in Figure 2): ${\omega }_1=({\overline{x}}_0,{\overline{x}}_1,{\overline{x}}_2^1,{\overline{x}}_3,{\overline{x}}_4^2,{\overline{x}}_5,{\overline{x}}_6)$, ${\omega }_2=({\overline{x}}_0,{\overline{x}}_1,x_2^2,z_3)$, ${\omega }_3=({\overline{x}}_0,{\overline{x}}_1,{\overline{x}}_2^1,{\overline{x}}_3,x_4^1)$ and ${\omega }_4=({\overline{x}}_0,{\overline{x}}_1,{\overline{x}}_2^1,{\overline{x}}_3,x_4^3,z_7)$. Note that players use the ranks when making decision at node ${\overline{x}}_5$.

To demonstrate the implementation of the incremental IDP and its properties we will adopt the Shapley value as a single valued cooperative solution. The values of the γ-characteristic function $V^{x_0}$ for the original game ${\mathrm{\Gamma }}^{x_0}(N,V^{x_0})$ and the Shapley value ${\phi }^{x_0}$ are

Consider, for instance, the incremental IDP along the longest cooperative trajectory ${\omega }_2=({\overline{x}}_0,\dots ,{\overline{x}}_6)$ from $\mathrm{\Omega }\left(\overline{u}\right)$. If we calculate γ-characteristic functions using Attitude SPE algorithm for the subgames, we get the following results.

Subgame ${\mathrm{\Gamma }}^{{\overline{x}}_1}(N,V^{{\overline{x}}_1})$:

Subgame ${\mathrm{\Gamma }}^{{\overline{x}}_2^1}(N,V^{{\overline{x}}_2^1})$:

Subgame ${\mathrm{\Gamma }}^{x_2^2}(N,V^{x_2^2})$:

Subgame ${\mathrm{\Gamma }}^{{\overline{x}}_3}(N,V^{{\overline{x}}_3})$:

Subgame ${\mathrm{\Gamma }}^{{\overline{x}}_4^2}(N,V^{{\overline{x}}_4^2})$:

Subgame ${\mathrm{\Gamma }}^{x_4^3}(N,V^{x_4^3})$:

Subgame ${\mathrm{\Gamma }}^{{\overline{x}}_5}(N,V^{{\overline{x}}_5})$:

Finally, ${\phi }^{{\overline{x}}_6}=h^{{\overline{x}}_6}=(12,0,0)$.

One can calculate the incremental IDP $\left\{{\beta }_i\left({\overline{x}}_{\tau }\right),{\overline{x}}_{\tau }\in {\omega }_2\right\}$ using (36) and (37):

	${\overline{x}}_0$	${\overline{x}}_1$	${\overline{x}}_2^1$	${\overline{x}}_3$	${\overline{x}}_4^2$	${\overline{x}}_5$	${\overline{x}}_6$
${\beta }_1\left({\overline{x}}_{\tau }\right)$	6	0	$-10$	12	36	$-12$	12
${\beta }_2\left({\overline{x}}_{\tau }\right)$	0	6	4	0	$-6$	12	0
${\beta }_3\left({\overline{x}}_{\tau }\right)$	0	0	18	0	6	24	0

,

Note that the subgame consistency conditions at nodes ${\overline{x}}_1$, ${\overline{x}}_3$ and ${\overline{x}}_5$ according to (32)–(34) respectively take the form:

$${\beta }_i\left({\overline{x}}_0\right)+{\phi }_i^{{\overline{x}}_1}={\phi }_i^{x_0},i\in N,or\left(\begin{array}{c}6\\ 0\\ 0\end{array}\right)+\left(\begin{array}{c}19\frac{1}{6}\\ 17\frac{1}{6}\\ 25\frac{2}{3}\end{array}\right)=\left(\begin{array}{c}25\frac{1}{6}\\ 17\frac{1}{6}\\ 25\frac{2}{3}\end{array}\right),$$

$${\beta }_i\left({\overline{x}}_2^1\right)+{\phi }_i^{{\overline{x}}_3}={\phi }_i^{x_2^1},i\in N,or\left(\begin{array}{c}-10\\ 4\\ 18\end{array}\right)+\left(\begin{array}{c}31\\ 13\\ 20\end{array}\right)=\left(\begin{array}{c}21\\ 17\\ 38\end{array}\right),$$

$${\beta }_i\left({\overline{x}}_4^2\right)+{\phi }_i^{{\overline{x}}_5}={\phi }_i^{x_4^2},i\in N,or\left(\begin{array}{c}36\\ -6\\ 6\end{array}\right)+\left(\begin{array}{c}0\\ 12\\ 24\end{array}\right)=\left(\begin{array}{c}36\\ 6\\ 30\end{array}\right).$$

It is known that the classical incremental IDP for multistage (and differential) games may imply negative current payments to some players at some positions (see References [4,10,17,38] for details). As one can observe in Ex. 2, this drawback of the incremental IDP may appear in the extensive-form game with chance moves as well. Two approaches how to overcome this possible disadvantage were suggested in References [4,10]. Unfortunately, as it was firstly proved in Reference [10], in general it is impossible to design a time consistent IDP which satisfies both the balance condition and non-negativity constraint.

Proposition 3.

The incremental IDP (36), (37) satisfies strict balance condition (28), the subgame efficiency condition (27), and the subgame consistency conditions (32)–(35).

Proof. 

Incremental IDP $\beta$ was proved to satisfiy strict balance condition (28) in Reference [9]. The proof of subgame consistency can be carried out by direct verification. For instance, consider the case when $t\left(1\right)+1<t⩽t\left(2\right)$. Then, using Remark 2 we get

$$\sum _{\tau =t\left(1\right)+1}^{t-1}{\beta }_i\left({\overline{x}}_{\tau }\right)=\left({\phi }_i^{{\overline{x}}_{t\left(1\right)+1}}-{\phi }_i^{{\overline{x}}_{t\left(1\right)+2}}\right)+\dots +\left({\phi }_i^{{\overline{x}}_{t-1}}-{\phi }_i^{{\overline{x}}_t}\right)={\phi }_i^{{\overline{x}}_{t\left(1\right)+1}}-{\phi }_i^{{\overline{x}}_t}.$$

Obviously, (33) is satisfied.

The proof that IDP (36), (37) satisfies subgame efficiency (27) is based on direct calculations but rather cumbersome in general case (i.e., for arbitrary game ${\mathrm{\Gamma }}^{x_0}$). Let us demonstrate how it works for the game in Example 2. For instance we verify that the incremental IDP meets the subgame efficiency condition at node ${\overline{x}}_3$.

Note that $\mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_3}\right)=\left\{{\omega }_1^{{\overline{x}}_3}=({\overline{x}}_3,x_4^1);{\omega }_2^{{\overline{x}}_3}=({\overline{x}}_3,{\overline{x}}_4^2,{\overline{x}}_5,{\overline{x}}_6);{\omega }_3^{{\overline{x}}_3}=({\overline{x}}_3,x_4^3,z_7)\right\}$ while $p({\omega }_1^{{\overline{x}}_3},{\overline{u}}^{{\overline{x}}_3})=\pi \left(x_4^1\right|{\overline{x}}_3)$, $p({\omega }_2^{{\overline{x}}_3},{\overline{u}}^{{\overline{x}}_3})=\pi \left({\overline{x}}_4^2\right|{\overline{x}}_3)$ and $p({\omega }_3^{{\overline{x}}_3},{\overline{u}}^{{\overline{x}}_3})=\pi \left(x_4^3\right|{\overline{x}}_3)$. Then, using (32), (33), Remark 2, equality ${\displaystyle \sum _{x_4^k\in S\left({\overline{x}}_3\right)}}\pi \left(x_4^k\right|{\overline{x}}_3)=1$ and the notation

$${\mathrm{\Phi }}_i^4=\sum _{k=1}^3\pi \left(x_4^k\right|{\overline{x}}_3)·{\phi }_i^{x_4^k},$$

we obtain

$$\begin{array}{c}\hfill {\displaystyle \sum _{{\omega }_k^{{\overline{x}}_3}\in \mathrm{\Omega }\left({\overline{u}}^{{\overline{x}}_3}\right)}}p({\omega }_k^{{\overline{x}}_3},{\overline{u}}^{{\overline{x}}_3})·\sum _{\tau =3}^{T\left(k\right)}{\beta }_i\left({\overline{x}}_{\tau }\right)=\pi \left(x_4^1\right|{\overline{x}}_3)·\left[\left({\phi }_i^{{\overline{x}}_3}-{\mathrm{\Phi }}_i^4\right)+{\phi }_i^{x_4^1}\right]+\\ \hfill +\pi \left({\overline{x}}_4^2\right|{\overline{x}}_3)·\left[\left({\phi }_i^{{\overline{x}}_3}-{\mathrm{\Phi }}_i^4\right)+\left({\phi }_i^{{\overline{x}}_4^2}-{\phi }_i^{{\overline{x}}_5}\right)+\left({\phi }_i^{{\overline{x}}_5}-{\phi }_i^{{\overline{x}}_6}\right)+{\phi }_i^{{\overline{x}}_6}\right]+\\ \hfill +\pi \left({\overline{x}}_4^3\right|{\overline{x}}_3)·\left[\left({\phi }_i^{{\overline{x}}_3}-{\mathrm{\Phi }}_i^4\right)+\left({\phi }_i^{x_4^3}-{\phi }_i^{z_7}\right)+{\phi }_i^{z_7}\right]={\phi }_i^{{\overline{x}}_3}·\sum _{k=1}^3\pi \left(x_4^k\right|{\overline{x}}_3)+\\ \hfill +\pi \left(x_4^1\right|{\overline{x}}_3)·\left[-{\mathrm{\Phi }}_i^4+{\phi }_i^{x_4^1}\right]+\pi \left({\overline{x}}_4^2\right|{\overline{x}}_3)·\left[-{\mathrm{\Phi }}_i^4+{\phi }_i^{{\overline{x}}_4^2}\right]+\pi \left(x_4^3\right|{\overline{x}}_3)·\left[-{\mathrm{\Phi }}_i^4+{\phi }_i^{x_4^3}\right]=\\ \hfill ={\phi }_i^{{\overline{x}}_3}-{\mathrm{\Phi }}_i^4·\sum _{k=1}^3\pi \left(x_4^k\right|{\overline{x}}_3)+\sum _{k=1}^3\pi \left(x_4^k\right|{\overline{x}}_3)·{\phi }_i^{x_4^k}={\phi }_i^{{\overline{x}}_3}.\end{array}$$

□

According to Proposition 3, the incremental payment schedule (36), (37) can be used to implement a long-term cooperative agreement in an extensive-form game with chance moves.

6. Conclusions

In the paper we purposes to design a mechanism of the players’ sustainable long-term cooperation that satisfies a number of good properties. To this aim we formalised the players’ rank based algorithm for selecting a unique optimal bundle of cooperative trajectories, and proved that corresponding cooperative strategy profile satisfies time consistency. To calculate $\gamma$-characteristic function one need to have a specific method for constructing a unique (subgame perfect) equilibrium at any extensive-form game with chance moves. Hence, we formalised a backwards induction procedure refinement based on the players’ attitude vectors—the so-called attitude SPE algorithm.

As a result of reexamination of the “IDP time consistency in the whole game” concept, we suggest to adopt the concept of subgame consistency, introduced in Reference [36] for differential stochastic games and then extend it to dynamic stochastic games in References [37,38]. The definition of subgame consistency for extensive-form game with chance moves is provided. This property takes into account such an interesting feature of the games under consideration that when the players make a decision in the subgame ${\mathrm{\Gamma }}^{x_t}$ after the chance move occurs, they need to recalculate their expected optimal payoffs-to-go since the original optimal bundle of cooperative trajectories shrinks after each chance node. It is worth noting that a similar approach, based on the IDP subgame consistency notion could be applied to dynamic games played over event trees ([14,16,20]). We proved that the incremental IDP specified for multistage games with chance moves in Reference [9] satisfy subgame consistency and subgame efficiency as well as the strict balance condition.

It follows from Propositions 1–3 that two specified algorithms combined with the $\gamma$-characteristic function, and the incremental payment schedule together constitute a mechanism of the players’ sustainable cooperation that satisfies a number of good properties and could be used in extensive-form games with chance moves. Note that the main result of the paper—Proposition 3—does not depend on the specific method which the players employ to calculate the characteristic function as well as on the specific single-valued cooperative solution meeting (25) and (26).

Since this is the first time that subgame consistent solutions are examined for extensive-form games with chance moves, further research along this line is expected. It is surely of interest to develop appropriate software application to implement proposed algorithms in arbitrary extensive-form game with chance moves. Possibly, one can use the so-called Game Theory Explorer [30] when developing such software tools for 2-person extensive games. Further, it might be interesting to run experiments with large-scale datasets, after the software application that allows to construct unique SPE, the optimal bundle of cooperative trajectories, $\gamma$-characteristic function, and so forth, will be developed.

Let us notice some preliminary suggestions on how one can use such software application to run simulations. First, one can vary the main parameter—the length of the game tree, and the additional parameters such as the game structure, the players’ payoffs, probabilities of transitions, and so forth, to obtain practical estimations of the proposed algorithms complexity and scalability. Secondarily, one can generate external disturbances of the stage payoffs and probabilities and vary the players’ attitude vectors to carry out the sensitivity analysis of the proposed non-cooperative and cooperative solutions. Further, it is of interest to get experimental estimations of the price of anarchy and the price of stability for the class of games under consideration. Finally, one can use such software application to check whether the additional properties (non-negativity, irrational-behavior-proof conditions, etc.) of the proposed incremental IDP and other payment schedules (see, e.g., Reference [15]) are satisfied for given extensive-form game with chance moves.