On a Simplified Method of Defining Characteristic Function in Stochastic Games

Abstract

In the paper, we propose a new method of constructing cooperative stochastic game in the form of characteristic function when initially non-cooperative stochastic game is given. The set of states and the set of actions for any player is finite. The construction of the characteristic function is based on a calculation of the maximin values of zero-sum games between a coalition and its anti-coalition for each state of the game. The proposed characteristic function has some advantages in comparison with previously defined characteristic functions for stochastic games. In particular, the advantages include computation simplicity and strong subgame consistency of the core calculated with the values of the new characteristic function.

1. Introduction

When a non-cooperative game is initially defined, the problem of construction of a cooperative version of the game is actual if players start acting as a unique coalition to maximize their joint payoff or minimize joint costs. The classical approach is to define cooperative game in a form of characteristic function that assigns the value for any coalition of players. Subsequently, based on this function one can calculate the imputation of the joint payoff allocating it among players. The component of the imputations may vary if we calculate them based on different characteristic functions. Therefore, the way of defining this function is important and it has influence on the players’ payoffs in cooperative game. Moreover, some approaches to define characteristic function make it impossible to apply in dynamic or differential games because of computational difficulties. Additionally, the way of constructing characteristic function also influences on the consistency properties of cooperative solutions that are realized in dynamics.

The choice of the approach on how to define characteristic function also depends on the background of the considered problem if it arises from an applied area. The existence and uniqueness issues are also actual when one chooses the way of constructing characteristic function. There exist different approaches that can be applied to stochastic game. The so-called maxmin and minmax approaches define the value of the function for coalition S as maxmin and minmax payoff of coalition S in zero-sum game against coalition of all left-out players [1,2]. Another approach is proposed in [3,4] when the value of coalition S is defined as its payoff in the Nash equilibrium in the non-cooperative game between coalition S and left-out players acting individually. The calculation of characteristic function in two-step procedure is proposed in [5], in which the authors first find an n-player non-cooperative equilibrium and then allow coalition S to optimize its payoff, assuming that left-out players use their Nash equilibrium actions found at the first step. The properties of this function are examined in [6,7]. Another two-stage approach for defining characteristic function is proposed in [8], in which the strategies maximizing total payoff of the players are first found. Subsequently, these strategies are used by the players from coalition S, while the out-coalition players use the strategies minimizing the total payoff of players from S. The joint payoff of players from the coalition equals the value of characteristic function for this coalition.

The new simplified method of constructing characteristic function in multistage games is introduced in [9]. They examine the properties of this function and proved that the corresponding core is strongly subgame-consistent in multistage game. This property cannot be proved in general case when the characteristic function is constructed with the classical approaches, like maxmin or minmax.

In the paper, we adopt the method of constructing the characteristic function proposed in [9] to stochastic games. Based on the values of the characteristic function, one can determine the core. Moreover, the core satisfies the strong subgame consistency property, which is a refinement of subgame consistency on the case of set-valued cooperative solutions. The problem of subgame consistency is originally examined for differential games in [10,11]. The construction of a special payment scheme, called imputation distribution procedure (see [11]), allows for coping with the problem of time inconsistency of cooperative solutions. This problem is described for stochastic games in [12,13,14] in the case of unique-valued cooperative solutions. The node-consistent core is constructed in dynamic games played over event trees in [15]. The strong subgame consistency of the set-valued cooperative solution, like the core, guarantees players to obtain, in total, the solution from initially defined core. It means that, in any intermediate time period, the solution is the sum of obtained payments up to the current period, and the core elements of subgame starting from the next time period. The strong subgame consistency condition is proposed in [16]. The subcore satisfying strong subgame property is constructed for multistage games in [17]. The problem of subgame consistency is actual for different classes of dynamic and differential games and it is examined in [18] for stochastic games with finite duration, in [19] for differential games with finite time horizon, in [20] for multistage games. In the paper, we construct characteristic function for stochastic game in a special way and calculate the core while using the values of this function. The core satisfies strong subgame consistency property. To prove this result, we define the imputation distribution procedure, which determines the payments to the players in any state realized in the game process.

The rest of the paper is organized, as follows. We describe the model of stochastic games in Section 2.1. In Section 2.2, we define the new approximated characteristic function for stage games, and then extend this approach to the case of stochastic game in Section 2.3. We formulate the definition of the imputation distribution procedure for stochastic games and describe the idea of strongly subgame consistency of the core in Section 3. We briefly conclude in Section 4.

2. Cooperative Stochastic Games

2.1. Model

Consider a non-cooperative stochastic game G given by

$$G=\left(N,\Omega ,{\{\Gamma \left(\omega \right)\}}_{\omega \in \Omega },{\pi }_0,{\left\{p\left({\omega }^{″}\right|{\omega }^{\prime },a^{{\omega }^{\prime }})\right\}}_{\begin{array}{c}{\omega }^{\prime },{\omega }^{″}\in \Omega \\ a^{{\omega }^{\prime }}\in {\prod }_{i\in N}{A}_i^{{\omega }^{\prime }}\end{array}},\delta \right),$$

(1)

where

• $N=\{1,\dots ,n\}$ is the set of players.
• $\Omega =\{{\omega }_1,\dots ,{\omega }_k\}$ is the finite set of states.
• $\Gamma \left(\omega \right)$ is the game in normal form associated with state $\omega$. The set of players N is common for any state $\omega$. Let $A_i^{\omega }$ be a finite set of actions of player $i\in N$ in state $\omega$, $a_i^{\omega }\in A_i^{\omega }$ be an action of player $i\in N$ in this state; $K_i^{\omega }:{\prod }_{j\in N}{A}_j^{\omega }\to \mathbb{R}$ be a payoff function of player i in state $\omega$.
• $p(·|\omega ,a^{\omega }):\Omega \times A^{\omega }\to \Delta (\Omega )$ is a transition function from state $\omega$ when action profile $a^{\omega }\in {\prod }_{j\in N}{A}_j^{\omega }$ is realized, where $\Delta (\Omega )$ is a probability distribution over set $\Omega$.
• ${\pi }_0=({\pi }_0^{{\omega }_1},\dots ,{\pi }_0^{{\omega }_k})$ is an initial state distribution.
• $\delta \in (0,1)$ is a common discount factor.

Denote by $G^{\omega }$ the subgame of G starting from state $\omega$ defined by (1) with ${\pi }_0$, such that ${\pi }_0^{\omega }=1$ and ${\pi }_0^{{\omega }^{\prime }}=0$ for any state ${\omega }^{\prime }\ne \omega$.

We assume that, in stochastic game G, the set of any player’s strategies $H_i$ is stationary. The stationary strategy of player i is ${\eta }_i$ assigning action (maybe mixed) $a_i\in \Delta \left(A_i^{\omega }\right)$ to any state $\omega$. The vector $({\eta }_1,\dots ,{\eta }_n)\in {\prod }_{j\in N}{H}_j$ is a stationary strategy profile in stochastic game G. It is obvious that a stationary strategy ${\eta }_i$ of player $i\in N$ in game G is the stationary strategy of this player in any subgame $G^{\omega }$.

By the payoff of player i, we assume the expected payoff in stochastic subgame $G^{\omega }$ given by

$$E_i^{\omega }\left(\eta \right)=K_i^{\omega }\left(a^{\omega }\right)+\delta \sum _{{\omega }^{\prime }\in \Omega }p\left({\omega }^{\prime }\right|\omega ,a^{\omega })E_i^{{\omega }^{\prime }}\left(\eta \right).$$

(2)

where $\eta \in H={\prod }_{j\in N}{H}_j$ is a stationary strategy profile such that $\eta \left(\omega \right)=a^{\omega }\in {\prod }_{j\in N}{A}_j^{\omega }$. We rewrite Equation (2) in a vector form and obtain

$$E_i\left(\eta \right)=K_i\left(a\right)+\delta \Pi \left(\eta \right)E_i\left(\eta \right),$$

(3)

where $E_i\left(\eta \right)={(E_i^{{\omega }_1}\left(\eta \right),\dots ,E_i^{{\omega }_k}\left(\eta \right))}^{\prime }$, $K_i\left(a\right)={(K_i^{{\omega }_1}\left(a^{{\omega }_1}\right),\dots ,K_i^{{\omega }_k}\left(a^{{\omega }_k}\right))}^{\prime }$. A matrix of transition probabilities is formed in the following way

$$\Pi \left(\eta \right)=\left(\begin{array}{ccc}p\left({\omega }_1\right|{\omega }_1,a^{{\omega }_1})& \dots & p\left({\omega }_k\right|{\omega }_1,a^{{\omega }_1})\\ p\left({\omega }_1\right|{\omega }_2,a^{{\omega }_2})& \dots & p\left({\omega }_k\right|{\omega }_2,a^{{\omega }_2})\\ \dots & \dots & \dots \\ p\left({\omega }_1\right|{\omega }_k,a^{{\omega }_k})& \dots & p\left({\omega }_k\right|{\omega }_k,a^{{\omega }_k})\end{array}\right)$$

(4)

in which each row contains transition probabilities from a corresponding state.

Equation (3) implies the explicit formula to calculate the expected payoff of player i when the stationary strategy profile $\eta$ is realized:

$$E_i\left(\eta \right)={\left({\mathbb{I}}_k-\delta \Pi \left(\eta \right)\right)}^{-1}{K}_i\left(a\right),$$

where ${\mathbb{I}}_k$ is an identity matrix of size $k\times k$. Inverted matrix ${\left({\mathbb{I}}_k-\delta \Pi \left(\eta \right)\right)}^{-1}$ always exists for discount factor $\delta \in (0,1)$.

Taking into account the probability distribution ${\pi }_0$, we calculate the expected payoff in game G, as

$${\overline{E}}_i\left(\eta \right)={\pi }_0{E}_i\left(\eta \right)={\pi }_0{\left({\mathbb{I}}_k-\delta \Pi \left(\eta \right)\right)}^{-1}{K}_i\left(a\right).$$

(5)

If players cooperate, they find the cooperative strategy profile ${\eta }^{*}$ maximizing the total expected payoff, which is

$${\eta }^{*}=arg\underset{\eta \in H}{max}\sum _{i\in N}{\overline{E}}_i\left(\eta \right).$$

We should notice that ${\eta }^{*}$ is a pure stationary strategy profile. The profile ${\eta }^{*}$ is such that ${\eta }_i^{*}\left(\omega \right)=a_i^{\omega *}\in A_i^{\omega }$, $\omega \in \Omega$. We also assume that the profile ${\eta }^{*}$ is such that ${max}_{\eta \in H}{\sum }_{i\in N}{E}_i^{\omega }\left(\eta \right)={\sum }_{i\in N}{E}_i^{\omega }\left({\eta }^{*}\right)$ for any state $\omega \in \Omega$, which means that the cooperative strategy profile maximizes the total payoff of the players independently of which state is initial. This assumption is usually satisfied for most stochastic games.

To define cooperative game when the non-cooperative stochastic game is given, we use the classical approach and define it in the form of characteristic function $v:2^N\to {\mathbb{R}}^1$ whose values estimate the “power” of any coalition or the subset of players. In [21], the characteristic function value for coalition S in subgame starting at any state $\omega$ is defined in as maxmin value, which is

$$v^{\omega }\left(S\right)=val{G}_S^{\omega },$$

(6)

where $G_S^{\omega }$ is a zero-sum stochastic subgame starting at state $\omega$, in which coalition S is a maximizing player, coalition $N\backslash S$ is a minimizing player. Existence of the value of game $G_S^{\omega }$ for stochastic games is proved in [22].

2.2. Approximated Characteristic Function for State Games

Before we define a characteristic function in a new form, we need to make additional calculations. First, we consider state games and propose a scheme of calculation of the approximated characteristic function values for any state. Define characteristic function for a state $\omega \in \Omega$ or one-shot game $\Gamma \left(\omega \right)$ given in normal form while using the maxmin approach:

$$v(\omega ,S)=\underset{a_S\in {\displaystyle \prod _{j\in S}}{A}_j^{\omega }}{max}\underset{a_{N\backslash S}\in {\displaystyle \prod _{j\in N\backslash S}}{A}_j^{\omega }}{min}\sum _{i\in S}{K}_i^{\omega }(a_S^{\omega },a_{N\backslash S}^{\omega }),$$

(7)

where maxmin in (7) is found in pure strategies.

Let $C\left(\omega \right)$ be a non-empty core in the game defined in state $\omega$ using c.f. (7), which is

$$C\left(\omega \right)=\left\{({\alpha }_1\left(\omega \right),\dots ,{\alpha }_n\left(\omega \right)):\sum _{i\in S}{\alpha }_i\left(\omega \right)⩾v(\omega ,S),\forall S\subset N,\sum _{i\in N}{\alpha }_i\left(\omega \right)=v(\omega ,N)\right\}$$

(8)

Remark 1.

We assume that conditions under which the core $C\left(\omega \right)$ exists for any state ω are satisfied. The core $C\left(\omega \right)$ is non-empty if and only if for any function $\psi :2^N\backslash ⌀\to [0,1]$, where ${\sum }_{S\in 2^N:S\ni i}\psi \left(S\right)=1$ for any $i\in N$, condition (see [23,24])

$$\sum _{S\in 2^N\backslash ⌀}\psi \left(S\right)v(\omega ,S)\le v(\omega ,N)$$

(9)

holds. Characteristic function $v(\omega ,S)$ is defined by (7). We refer to the book [25] for further discussion of non-emptiness of the core.

Second, for any coalition $S\subseteq N$ define maximal value of characteristic function (7) over set $\Omega$:

$$\widehat{w}\left(S\right)=\underset{\omega \in \Omega }{max}v(\omega ,S),$$

(10)

which is the maximal value that coalition S can obtain in state games.

The next step is to define the approximated value of the characteristic function for any state in the following way. Let for any state $\omega \in \Omega$ the approximated characteristic function $w(\omega ,S)$ be given as

$$w(\omega ,S)=\left\{\begin{array}{cc}{\displaystyle \sum _{i\in S}}{K}_i^{\omega }\left(a^{\omega *}\right),\hfill & \mathrm{if}S=N,\hfill \\ \widehat{w}\left(S\right),\hfill & \mathrm{if}S\ne N.\hfill \end{array}\right.$$

(11)

In Equation (11), the summarized payoff of the players adopting cooperative action profile $a^{\omega *}$ is assigned to the grand coalition. The approximated (maximal possible value over all possible states) values of characteristic function $\widehat{w}\left(S\right)$ given by (10) are assigned to any coalition S different from N. Denote the core constructed with the values of characteristic function (11) as $D\left(\omega \right)$ and assume that it is non-empty for any state $\omega$,

$$D\left(\omega \right)=\left\{({\alpha }_1\left(\omega \right),\dots ,{\alpha }_n\left(\omega \right)):\sum _{i\in S}{\alpha }_i\left(\omega \right)⩾w(\omega ,S),\forall S\subset N,\sum _{i\in N}{\alpha }_i\left(\omega \right)=w(\omega ,N)\right\}.$$

(12)

Lemma 1.

Let for any coalition $S\subset N$, $S\ne N$, the inequality $\widehat{w}\left(S\right)<\underset{\omega \in \Omega }{min}v(\omega ,N)$ hold. If condition

$$\sum _{i\in N}{K}_i^{\omega }\left(a^{\omega *}\right)=\underset{a^{\omega }\in {\displaystyle \prod _{j\in N}}{A}_j^{\omega }}{max}\sum _{i\in N}{K}_i^{\omega }\left(a^{\omega }\right),$$

(13)

is true, and the core $D\left(\omega \right)$ is non-empty for any ω, and then $D\left(\omega \right)\subset C\left(\omega \right)$.

Proof. 

If there exists coalition $S\subset N$, $S\ne N$, such that $\widehat{w}\left(S\right)⩾\underset{\omega \in \Omega }{min}v(\omega ,N)$, then the core $D\left(\omega \right)$ is empty. Assuming the non-emptiness of the core $D\left(\omega \right)$, we consider any imputation $\alpha \left(\omega \right)\in D\left(\omega \right)$. If condition (13) is true, it means that ${\displaystyle \sum _{i\in N}}{\alpha }_i\left(\omega \right)=v(\omega ,N)=w(\omega ,N)$.

Subsequently, for any coalition $S\subset N$, we have ${\displaystyle \sum _{i\in S}}{\alpha }_i\left(\omega \right)⩾w(\omega ,S)=\widehat{w}\left(S\right)=\underset{\omega \in \Omega }{max}v(\omega ,S)⩾v(\omega ,S)$, which proves that $\alpha \left(\omega \right)\in C\left(\omega \right)$. □

Remark 2.

Condition (13) states that the maximal total payoff of the players in state ω coincides with their payoff if players adopt actions prescribed by the cooperative strategy profile. It may not be satisfied in general case in dynamic games. If condition (13) is not true, the main result of the paper can be proved, but it requires a modification in the method of characteristic function definition. We leave this case for future research.

Remark 3.

We assume that the approximated core $D\left(\omega \right)$ is non-empty for any ω. The conditions under which it is non-empty are similar to the ones given in Remark 1, but in Equation (9) characteristic function $w(\omega ,S)$ given by (11) is used. If the conditions of Lemma 1 are satisfied, then $D\left(\omega \right)\subset C\left(\omega \right)$, and non-emptiness of approximated core $D\left(\omega \right)$ implies non-emptiness of core $C\left(\omega \right)$.

Example 1.

Consider three-player stochastic game with two states (${\omega }_1$ and ${\omega }_2$). The sets of actions of player 1, 2, and 3 in state ${\omega }_1$ (${\omega }_2$) are $\{a_1,a_2\}$, $\{b_1,b_2\}$ and $\{c_1,c_2\}$ ($\{{\alpha }_1,{\alpha }_2\}$, $\{{\zeta }_1,{\zeta }_2\}$, $\{{\gamma }_1,{\gamma }_2\}$), respectively. The payoff functions are given by the following matrices:

• in state ${\omega }_1$:

  $$c_1:\begin{array}{ccc}& b_1& b_2\\ \begin{array}{c}{a}_1\\ a_2\end{array}& (\begin{array}{c}(10,10,8)\\ (15,0,0)\end{array}& \begin{array}{c}(0,15,0)\\ (5,5,5)\end{array})\end{array}{c}_2:\begin{array}{ccc}& b_1& b_2\\ \begin{array}{c}{a}_1\\ a_2\end{array}& (\begin{array}{c}(0,0,15)\\ (4,4,2)\end{array}& \begin{array}{c}(2,4,4)\\ (0,0,0)\end{array})\end{array}$$
• in state ${\omega }_2$:

  $${\gamma }_1:\begin{array}{ccc}& {\zeta }_1& {\zeta }_2\\ \begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}& (\begin{array}{c}(2,1,1)\\ (0,4,2)\end{array}& \begin{array}{c}(4,0,2)\\ (7,5,3)\end{array})\end{array}{\gamma }_2:\begin{array}{ccc}& {\zeta }_1& {\zeta }_2\\ \begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}& (\begin{array}{c}(2,3,0)\\ (3,4,3)\end{array}& \begin{array}{c}(4,2,4)\\ (7,5,7)\end{array})\end{array}$$

Player 1 chooses a row, player 2 chooses a column and player 3 chooses a matrix.

The transition probabilities are written in the matrices:

• for state ${\omega }_1$:

  $$c_1:\begin{array}{ccc}& b_1& b_2\\ \begin{array}{c}{a}_1\\ a_2\end{array}& (\begin{array}{c}(0.5,0.5)\\ (0,1)\end{array}& \begin{array}{c}(0,1)\\ (0,1)\end{array})\end{array}{c}_2:\begin{array}{ccc}& b_1& b_2\\ \begin{array}{c}{a}_1\\ a_2\end{array}& (\begin{array}{c}(0,1)\\ (0.5,0.5)\end{array}& \begin{array}{c}(0.5,0.5)\\ (1,0)\end{array})\end{array}$$
• for state ${\omega }_2$:

  $${\gamma }_1:\begin{array}{ccc}& {\zeta }_1& {\zeta }_2\\ \begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}& (\begin{array}{c}(0,1)\\ (1,0)\end{array}& \begin{array}{c}(1,0)\\ (1,0)\end{array})\end{array}{\gamma }_2:\begin{array}{ccc}& {\zeta }_1& {\zeta }_2\\ \begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}& (\begin{array}{c}(0.2,0.8)\\ (0,1)\end{array}& \begin{array}{c}(0,1)\\ (1,0)\end{array})\end{array}$$
  The first (second) element in any entry of the matrix is the probability of transition from the particular state and action profile to state ${\omega }_1$ (state ${\omega }_2$). One can easily notice that the probabilistic transitions are defined in state ${\omega }_1$ when players choose action profiles $(a_1,b_1,c_1)$, $(a_2,b_1,c_2)$ and $(a_1,b_2,c_2)$, and in state ${\omega }_2$ when players choose action profiles $({\alpha }_1,{\zeta }_1,{\gamma }_2)$. All other transitions are deterministic.

The discount factor equals 0.9.

Cooperative strategy profile ${\eta }^{*}=({\eta }_1^{*},{\eta }_2^{*},{\eta }_3^{*})$ is such that

$${\eta }_1^{*}=(a_1,{\alpha }_2),{\eta }_2^{*}=(b_1,{\zeta }_2),{\eta }_2^{*}=(c_1,{\gamma }_2),$$

(14)

which prescribes any player to choose the first action in state ${\omega }_1$ and the second action in state ${\omega }_2$. The cooperative strategy profile defines a Markov chain with the structure that is depicted in Figure 1.

The players’ payoffs are $(10,10,8)$ in state ${\omega }_1$ and $(7,5,7)$ in state ${\omega }_2$. We obtain that the maximal total payoff of the players in state games coincide with the payoff that players get in states implementing cooperative strategy profile ${\eta }^{*}$. However, Theorem 1 is also true for the case when this condition is not satisfied.

First, we calculate the characteristic function $v(\omega ,S)$ by Equation (7) and its approximation $w(\omega ,S)$ by (11) for state games. The values of these functions are represented in

Table 1. Values of characteristic function v and approximated characteristic function w for states ${\omega }_1$ and ${\omega }_2$.

S	$\{1,2,3\}$	$\{1,2\}$	$\{1,3\}$	$\{2,3\}$	$\left\{1\right\}$	$\left\{2\right\}$	$\left\{3\right\}$
$v({\omega }_1,S)$	28	8	10	10	0	0	0
$v({\omega }_2,S)$	19	12	6	8	2	1	1
$w({\omega }_1,S)$	28	12	10	10	2	1	1
$w({\omega }_2,S)$	19	12	10	10	2	1	1

.

The cores of state games $C\left(\omega \right)$ and $D\left(\omega \right)$ calculated with values of functions $v(\omega ,S)$ and $w(\omega ,S)$ by Formulae (8) and (12) are non-empty for any ω and represented on Figure 2 and Figure 3 for ${\omega }_1$ and ${\omega }_2$ respectively.

2.3. New Approximated Characteristic Function for Stochastic Games

We propose a new method of determining characteristic function for stochastic games based on the values of approximated characteristic function defined in states and given by Formula (11).

We assume that coalition S at any state of the game may obtain $\widehat{w}\left(S\right)$ as maximum. Accordingly, this value is the maximal value that the coalition can get, regardless of the state that currently appears. If we summarize this value over infinite horizon with discount factor $\delta$, we can calculate the approximation or the upper bound of the payoff that coalition S can get in stochastic subgame starting from state $\omega$, which is

$$\overline{w}(\omega ,S)=\left\{\begin{array}{cc}\widehat{w}\left(S\right)+\delta \widehat{w}\left(S\right)+\dots ={\displaystyle \frac{1}{1-\delta }}\widehat{w}\left(S\right),\hfill & \mathrm{if}S\subset N,S\ne N,\hfill \\ {\displaystyle \sum _{i\in N}}{E}_i^{\omega }\left({\eta }^{*}\right),\hfill & \mathrm{if}S=N.\hfill \end{array}\right.$$

(15)

One should notice that, according to Equation (15), we save the value of characteristic function for grand coalition without approximation. The reason is that, when we define the allocation of a joint payoff, the players should redistribute the value that they obtain using cooperative strategy profile, but not the approximated one. The cooperative stochastic subgame is defined by the set of players N and function (15). In the following, we omit the set of players and refer the cooperative stochastic subgame as $\overline{w}(\omega ,S)$ given by (15).

Let $\overline{D}\left(\omega \right)$ be the core calculated with the values of function (15), i.e.,

$$\overline{D}\left(\omega \right)=\left\{({\alpha }_1\left(\omega \right),\dots ,{\alpha }_n\left(\omega \right)):\sum _{i\in S}{\alpha }_i\left(\omega \right)⩾\overline{w}(\omega ,S),\forall S\subset N,\sum _{i\in N}{\alpha }_i\left(\omega \right)=\overline{w}(\omega ,N)\right\}.$$

(16)

Let $\overline{D}\left(\omega \right)$ be non-empty for any $\omega \in \Omega$. We can compare the core $\overline{D}\left(\omega \right)$ constructed with the values of approximated function (15) and the core defined with the values of characteristic function defined with the classical approach. For any subgame $G^{\omega }$, we define characteristic function using the maxmin approach:

$$\overline{v}(\omega ,S)=\underset{{\eta }_S\in {\displaystyle \prod _{j\in S}}{H}_j}{max}\underset{{\eta }_{N\backslash S}\in {\displaystyle \prod _{j\in N\backslash S}}{H}_j}{min}\sum _{i\in S}{E}_i^{\omega }({\eta }_S,{\eta }_{N\backslash S}).$$

(17)

Let $\overline{C}\left(\omega \right)$ be a non-empty core of subgame $G^{\omega }$ constructed with the values of function (17).

Lemma 2.

Let for any coalition $S\subset N$, $S\ne N$ the inequality

$$\widehat{w}\left(S\right)<\underset{\omega \in \Omega }{min}v(\omega ,N)$$

(18)

hold, and $\overline{D}\left(\omega \right)$ is non-empty for any ω, then $\overline{D}\left(\omega \right)\subset \overline{C}\left(\omega \right)$.

Proof. 

If $\widehat{w}\left(S\right)<\underset{\omega \in \Omega }{min}v(\omega ,N)$ is not satisfied, then the core $\overline{D}\left(\omega \right)$ is empty by construction. Consider any imputation $\overline{\alpha }\left(\omega \right)\in \overline{D}\left(\omega \right)$ and prove that it belongs to the set $\overline{C}\left(\omega \right)$.

First, ${\displaystyle \sum _{i\in N}}{\overline{\alpha }}_i\left(\omega \right)=\overline{w}(\omega ,N)={\displaystyle \sum _{i\in N}}{E}_i^{\omega }\left({\eta }^{*}\right)=\overline{v}(\omega ,N)$.

Second, we prove that ${\displaystyle \sum _{i\in S}}{\overline{\alpha }}_i\left(\omega \right)⩾\overline{v}(\omega ,S)$ taking into account that ${\displaystyle \sum _{i\in S}}{\overline{\alpha }}_i\left(\omega \right)⩾\overline{w}(\omega ,S)$ for any $S\ne N$. We prove that $\overline{w}(\omega ,S)⩾\overline{v}(\omega ,S)$.

By definition, we have

$$\overline{v}(\omega ,S)=\underset{{\eta }_S}{max}\underset{{\eta }_{N\backslash S}}{min}{\displaystyle \sum _{i\in S}}{E}_i^{\omega }({\eta }_S,{\eta }_{N\backslash S}),$$

and we write the functional equation for the right-hand side of this equality and obtain the following

$$\begin{array}{cc}\hfill \underset{{\eta }_S}{max}\underset{{\eta }_{N\backslash S}}{min}{\displaystyle \sum _{i\in S}}{E}_i^{\omega }({\eta }_S,{\eta }_{N\backslash S})& =\underset{{\eta }_S}{max}\underset{{\eta }_{N\backslash S}}{min}\left\{{\displaystyle \sum _{i\in S}}{K}_i^{\omega }(a_S^{\omega },a_{N\backslash S}^{\omega })+\delta p(\omega ,a^{\omega }){\displaystyle \sum _{i\in S}}{E}_i({\eta }_S,{\eta }_{N\backslash S})\right\},\hfill \end{array}$$

where $p(\omega ,a^{\omega })$ is a vector $\left(p\left({\omega }^{\prime }\right|\omega ,a^{\omega }):{\omega }^{\prime }\in \Omega \right)$.

Let profile $({\eta }_S,{\eta }_{N\backslash S})$ be such that maxmin is reached at this profile, we can write the functional equation, as follows:

$$\begin{array}{cc}\hfill {\displaystyle \sum _{i\in S}}{E}_i^{\omega }({\eta }_S,{\eta }_{N\backslash S})& ={({\mathbb{I}}_k-\delta \Pi ({\eta }_S,{\eta }_{N\backslash S}))}^{-1}{\displaystyle \sum _{i\in S}}{K}_i(a_S,a_{N\backslash S})⩽\frac{1}{1-\delta }\underset{\omega \in \Omega }{max}\underset{a_S}{max}\underset{a_{N\backslash S}}{min}{\displaystyle \sum _{i\in S}}{K}_i^{\omega }(a_S^{\omega },a_{N\backslash S}^{\omega })\hfill \\ \hfill & =\frac{1}{1-\delta }\underset{\omega \in \Omega }{max}v(\omega ,S)=\overline{w}(\omega ,S).\hfill \end{array}$$

In the last inequality, we use the property of stochastic matrices, i.e., the sum of the elements in any row of matrix ${({\mathbb{I}}_k-\delta \Pi ({\eta }_S,{\eta }_{N\backslash S}))}^{-1}$ equal $1/(1-\delta )$, because $\Pi ({\eta }_S,{\eta }_{N\backslash S})$ is a stochastic matrix. The lemma is proved. □

Remark 4.

We assume non-emptiness of the approximated core $\overline{D}\left(\omega \right)$ in stochastic game with any initial state ω. If condition (18) in Lemma 2 is satisfied, the non-emptiness of the approximated cores $D\left(\omega \right)$ for any ω implies the non-emptiness of approximated core $\overline{D}\left(\omega \right)$. It follows from definition of characteristic function $\overline{w}(\omega ,S)$ and Formula (10). Moreover, the non-emptiness of approximated core $\overline{D}\left(\omega \right)$ implies non-emptiness of core $\overline{C}\left(\omega \right)$.

Example 2.

(continuation of Example 1) We continue calculations for stochastic game described in Example 1. Define characteristic function $\overline{v}$ by (17) and approximated characteristic function $\overline{w}$ by (15). The values of these functions are given in

Table 2. Values of characteristic function $\overline{v}$ and approximated characteristic function $\overline{w}$ for stochastic game starting from states ${\omega }_1$ and ${\omega }_2$.

S	$\{1,2,3\}$	$\{1,2\}$	$\{1,3\}$	$\{2,3\}$	$\left\{1\right\}$	$\left\{2\right\}$	$\left\{3\right\}$
$\overline{v}({\omega }_1,S)$	252.07	92.41	62.01	64.00	9.47	9.00	9.00
$\overline{v}({\omega }_2,S)$	245.86	95.17	77.87	60.00	10.52	10.00	10.00
$\overline{w}({\omega }_1,S)$	252.07	120.00	100.00	100.00	20.00	10.00	10.00
$\overline{w}({\omega }_2,S)$	245.86	120.00	100.00	100.00	20.00	10.00	10.00

.

The cores $\overline{C}\left(\omega \right)$ and $\overline{D}\left(\omega \right)$ constructed with the values of functions $\overline{v}$ and $\overline{w}$, respectively, are non-empty and depicted on Figure 4 and Figure 5 for initial states ${\omega }_1$ and ${\omega }_2$, respectively. One can notice that $\overline{D}\left(\omega \right)\subset \overline{C}\left(\omega \right)$ for any ω.

The approximated core $\overline{D}\left({\omega }_1\right)$ is defined as the set

$$\begin{array}{cc}\hfill \overline{D}\left({\omega }_1\right)=\{({\overline{\alpha }}_1,{\overline{\alpha }}_2,{\overline{\alpha }}_3):& {\overline{\alpha }}_1+{\overline{\alpha }}_2+{\overline{\alpha }}_3=252.07,{\overline{\alpha }}_1+{\overline{\alpha }}_2⩾120.00,{\overline{\alpha }}_1+{\overline{\alpha }}_3⩾100.00,\hfill \\ \hfill & {\overline{\alpha }}_2+{\overline{\alpha }}_3⩾100.00,{\overline{\alpha }}_1⩾20.00,{\overline{\alpha }}_2⩾10.00,{\overline{\alpha }}_3⩾10.00\}.\hfill \end{array}$$

The approximated core $\overline{D}\left({\omega }_2\right)$ is defined as the set

$$\begin{array}{cc}\hfill \overline{D}\left({\omega }_2\right)=\{({\overline{\alpha }}_1,{\overline{\alpha }}_2,{\overline{\alpha }}_3):& {\overline{\alpha }}_1+{\overline{\alpha }}_2+{\overline{\alpha }}_3=245.86,{\overline{\alpha }}_1+{\overline{\alpha }}_2⩾120.00,{\overline{\alpha }}_1+{\overline{\alpha }}_3⩾100.00,\hfill \\ \hfill & {\overline{\alpha }}_2+{\overline{\alpha }}_3⩾100.00,{\overline{\alpha }}_1⩾20.00,{\overline{\alpha }}_2⩾10.00,{\overline{\alpha }}_3⩾10.00\}.\hfill \end{array}$$

3. Strongly Subgame-Consistent Core in Stochastic Games

3.1. Imputation Distribution Procedure

In cooperation, players follow the cooperative strategy profile ${\eta }^{*}$ and then agree on the core as a cooperative solution of the game or the set of possible imputations of the joint payoff in the game. We assume that the core for any subgame $G^{\omega }$ is calculated based on function (15), which is $\overline{D}\left(\omega \right)$. Consider an imputation $\overline{\alpha }\left(\omega \right)\in \overline{D}\left(\omega \right)$. Obviously, if the players are paid step by step according to initially given payoff functions $K_i^{\omega }$, $i\in N$, we cannot guarantee that they will get the components of imputation $\overline{\alpha }\left(\omega \right)$ as an expected payoff in subgame $G^{\omega }$. Therefore, we define the scheme of state payments that, in total, will give the players to obtain the components of imputation $\overline{\alpha }\left(\omega \right)$.

Definition 1.

[10,11] We call the collection of vectors $({\beta }_i:i\in N)$, where ${\beta }_i=({\beta }_i\left({\omega }_1\right),\dots ,{\beta }_i\left({\omega }_k\right))$, ${\beta }_i\left(\omega \right)$ is a payment to player i in state ω in cooperative stochastic game, an imputation distribution procedure (IDP) of imputation $\overline{\alpha }\left(\omega \right)\in \overline{D}\left(\omega \right)$ if

• ${\displaystyle \sum _{i\in N}}{\beta }_i\left(\omega \right)={\displaystyle \sum _{i\in N}}{K}_i^{\omega }\left(a^{\omega *}\right)$ for any $\omega \in \Omega$;
• ${\overline{\alpha }}_i\left(\omega \right)=B_i^{\omega }$, where $B_i^{\omega }$ is the expected discounted sum of payments to player i in stochastic subgame starting from state ω, according to procedure β.

The expected sum of payments to player i made according to IDP can be calculated by formula (see [14]):

$$B_i^{\omega }={\pi }_0{(\mathbb{I}-\delta \Pi \left({\eta }^{*}\right))}^{-1}{\beta }_i,$$

where ${\pi }_0$ is such that ${\pi }_0^{\omega }=1$ and ${\pi }_0^{{\omega }^{\prime }}=0$ for any ${\omega }^{\prime }\ne \omega$.

Remark 5.

The IDP determined in Definition 1 for an imputation $\overline{\alpha }\left(\omega \right)\in \overline{D}\left(\omega \right)$ may be non-unique.

In the following section, we describe a property of the imputations from the core and corresponding IDP, which allows to narrow the set of IDP.

3.2. Strongly Subgame-Consistent Core

We formulate the property of strongly subgame consistency of the core and propose sufficient conditions of strongly subgame consistency of the core in stochastic games with characteristic function (15). We suppose that the cores of stochastic game G and any subgame $G^{\omega }$, $\omega \in \Omega$, are non-empty.

In cooperation, players agree on the joint implementation of cooperative strategy profile ${\eta }^{*}$ and expect to obtain the components of the imputation belonging to the core $\overline{D}\left(\omega \right)$ in the subgame stating from $\omega$. Reaching an intermediate state $\omega \in \Omega$, Player i chooses action $a_i^{\omega *}$ prescribed by cooperative strategy profile ${\eta }^{*}$ and gets payoff $K_i^{\omega }\left(a^{\omega *}\right)$. If the players recalculate the solution in the current subgame and find solution of cooperative subgame $G^{\omega }$, we would assume that the cooperative solution is chosen from the core $\overline{D}\left(\omega \right)$. It would be reasonable to require that the payoff received by a player in state $\omega$ summarized with the expected sum of any imputations from the cores $\overline{D}\left({\omega }^{\prime }\right)$, ${\omega }^{\prime }\in \Omega$, following state $\omega$, would be an imputation from the core $\overline{D}\left(\omega \right)$. If this property holds for any intermediate state $\omega \in \Omega$, then the core of cooperative stochastic game with characteristic function (15) is strongly subgame-consistent.

To determine a strongly subgame-consistent core, we need to define the so-called expected core at state $\omega$, i.e., we define the set of expected imputations belonging to the cores, which are cooperative solutions of the following subgames. We determine the expected core of state $\omega \in \Omega$, as follows:

$$E\overline{D}\left(\omega \right)=\left\{\delta \sum _{\omega \in \Omega }p\left({\omega }^{\prime }\right|\omega ,a^{\omega *})\overline{\alpha }\left({\omega }^{\prime }\right),\overline{\alpha }\left({\omega }^{\prime }\right)\in \overline{D}\left({\omega }^{\prime }\right)\right\}.$$

Definition 2.

We call the core $\overline{D}\left(\omega \right)$ strongly subgame consistent solution of cooperative stochastic game with approximated characteristic function $\overline{w}(\omega ,S)$ starting from state ω if for any imputation $\overline{\alpha }\left(\omega \right)\in \overline{D}\left(\omega \right)$ there exists an IDP $\beta =({\beta }_i:i\in N)$, where ${\beta }_i=({\beta }_i\left(\omega \right):\omega \in \Omega )$, satisfying condition:

$$\beta \oplus E\overline{D}\subset \overline{D},$$

(19)

where $E\overline{D}$ is the vector ${(E\overline{D}\left({\omega }_1\right),\dots ,E\overline{D}\left({\omega }_k\right))}^{\prime }$ of expected cores for states ${\omega }_1,\dots ,{\omega }_k$ respectively, $\overline{D}$ is a vector with elements which are sets, i.e., $\overline{D}={(\overline{D}\left({\omega }_1\right),\dots ,\overline{D}\left({\omega }_k\right))}^{\prime }$.

Remark 6.

The inclusion (19) is written in a vector form. To explain it, we write the first row of vector inclusion (19):

$$\beta \left({\omega }_1\right)\oplus E\overline{D}\left({\omega }_1\right)\subset \overline{D}\left({\omega }_1\right)$$

where $\beta \left({\omega }_1\right)\in {\mathbb{R}}^n$, $E\overline{D}\left({\omega }_1\right)\subset {\mathbb{R}}^n$, $\overline{D}\left({\omega }_1\right)\subset {\mathbb{R}}^n$. The operation $a\oplus C$, where $a\in {\mathbb{R}}^n$ and C is a set in ${\mathbb{R}}^n$, is defined as the set $\{a+c,forallc\in C\}$.

Theorem 1.

The core $\overline{D}\left(\omega \right)$, if it exists, is strongly subgame-consistent.

Proof. 

Following Definition 2 we need to prove that there exists an IDP of the elements from the core $\overline{D}\left(\omega \right)$ defined in (16) satisfying two properties from Definition 1, such that inclusion (19) is true.

Let for any imputation ${\overline{\alpha }}_i\left(\omega \right)\in \overline{D}\left(\omega \right)$, the IDP is calculated as

$${\beta }_i=({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right)){\overline{\alpha }}_i,$$

(20)

where ${\beta }_i={({\beta }_i\left({\omega }_1\right),\dots ,{\beta }_i\left({\omega }_k\right))}^{\prime }$ and ${\overline{\alpha }}_i={({\overline{\alpha }}_i\left({\omega }_1\right),\dots ,{\overline{\alpha }}_i\left({\omega }_k\right))}^{\prime }$.

First, we prove that $\beta$, defined in (20), satisfies properties 1 and 2 in Definition 1.

• Find the sum of ${\beta }_i$ over the set of players, we obtain

  $$\begin{array}{cc}\hfill {\displaystyle \sum _{i\in N}}{\beta }_i=({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right)){\displaystyle \sum _{i\in N}}{\overline{\alpha }}_i& =({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right)){(\overline{v}({\omega }_1,N),\dots ,\overline{v}({\omega }_k,N))}^{\prime }\hfill \\ \hfill & =({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right)){({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right))}^{-1}{\displaystyle \sum _{i\in N}}{K}_i\left(a^{*}\right)={\displaystyle \sum _{i\in N}}{K}_i\left(a^{*}\right),\hfill \end{array}$$

  or for any $\omega \in \Omega$ the equality ${\sum }_{i\in N}{\beta }_i\left(\omega \right)={\sum }_{i\in N}{K}_i^{\omega }\left(a^{\omega *}\right)$ is true.
• We prove that ${\overline{\alpha }}_i\left(\omega \right)=B_i^{\omega }$ or in vector form ${\overline{\alpha }}_i=B_i$, where $B_i={(B_i\left({\omega }_1\right),\dots ,B_i\left({\omega }_k\right))}^{\prime }$. We have

  $$B_i={({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right))}^{-1}{\beta }_i={({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right))}^{-1}({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right)){\overline{\alpha }}_i={\overline{\alpha }}_i.$$

Therefore, the payment vector ${\beta }_i$, $i\in N$, is the distribution procedure of imputation ${\overline{\alpha }}_i$.

Now, we prove that inclusion (19) holds. Let ${\beta }_i$ be given by Equation (20), there ${\overline{\alpha }}_i={({\overline{\alpha }}_i\left({\omega }_1\right),\dots ,{\overline{\alpha }}_i\left({\omega }_k\right))}^{\prime }$ and ${\overline{\alpha }}_i\left({\omega }_j\right)\in \overline{D}\left({\omega }_j\right)$ for any $j=1,\dots ,k$. Consider the sum $\beta \left(\omega \right)+\epsilon \left(\omega \right)$, where $\epsilon \left(\omega \right)$ is any vector from the expected core $E\overline{D}\left(\omega \right)$. Substituting expressions of ${\beta }_i$ from Equation (20) and element of the expected core into the sum, we get

$$\beta +\epsilon =({\mathbb{I}}_k-\delta \Pi \left({\eta }^{*}\right))\overline{\alpha }+\delta \Pi \left({\eta }^{*}\right)\overline{\alpha }=\overline{\alpha }\in \overline{D},$$

which proves the theorem. □

Theorem 1 gives the method of construction of payment scheme of any element from the core $\overline{D}$ defined by (16) while using values of function (15).

Example 3.

(continuation of Example 1 and 2) We demonstrate how to define IDP using a method from the proof of Theorem 1. Let for ${\omega }_1$ and ${\omega }_2$ the core imputations $\overline{\alpha }\left({\omega }_1\right)=(100.00,100.00,52.07)\in \overline{D}\left({\omega }_1\right)$ and $\overline{\alpha }\left({\omega }_2\right)=(50.00,95.86,100.00)\in \overline{D}\left({\omega }_2\right)$ be chosen. To calculate IDP by Formula (20), we need to define matrix $\Pi \left({\eta }^{*}\right)$, which is

$$\Pi \left({\eta }^{*}\right)=\left(\begin{array}{cc}0.5& 0.5\\ 0& 1\end{array}\right)$$

for cooperative strategy profile ${\eta }^{*}$ determined by (14).

Using Formula (20) with ${\overline{\alpha }}_1=(100.00,50.00)$, ${\overline{\alpha }}_2=(100.00,95.86)$, ${\overline{\alpha }}_3=(52.07,100.00)$, we obtain

$$\begin{array}{cc}\hfill {\beta }_1& =(32.50,-40.00),\hfill \\ \hfill {\beta }_2& =\left(\mathrm{11.86.5.86}\right),\hfill \\ \hfill {\beta }_3& =(-\mathrm{16.36.53.14}),\hfill \end{array}$$

where the first component of vector ${\beta }_i$ is the payment to player i in state ${\omega }_1$ and the second component is the payment in state ${\omega }_2$. We can easily check that collection of vectors $({\beta }_i:i\in N)$ satisfies conditions from Definition 1 of IDP.

The approximated cores $\overline{D}\left({\omega }_1\right)$ and $\overline{D}\left({\omega }_2\right)$ are strongly subgame-consistent, which is proved in Theorem 1.

Remark 7.

The new method of construction of the characteristic function or the so-called approximated characteristic function proposed in the paper allows not only to find the strongly subgame-consistent subset of the core, but also simplifies calculations. In the example, each player has two actions in any state. Therefore, he has four pure stationary strategies in a stochastic game, and there are 64 strategy profiles in the game. The calculations of maxmin payoff of a coalition in such games is a complicated computational problem. The new approach allows for avoiding these calculations using the values of approximated characteristic function defined in state games to determine the function for a stochastic game.

4. Conclusions

We have proposed a new method of constructing the characteristic function in stochastic games. The method simplifies calculations in comparison with the previously introduced approaches. An additional advantage of the method is that the core calculated with the values of this characteristic function satisfies strongly subgame consistency. This property positively characterizes the realization of the imputations from the core in a dynamic game process. The property of strongly subgame consistency is applied for set-valued cooperative solutions, like the core. We can briefly characterize the possible directions for future research in this area. We can also consider additional simplifications in characteristic function definitions, which allow not only to keep the strong subgame consistency properties of the core, but also to reduce the number of calculations defining cooperative stochastic game.