Discrete-Time Constrained Average Stochastic Games with Independent State Processes

Abstract

In this paper, we consider the discrete-time constrained average stochastic games with independent state processes. The state space of each player is denumerable and one-stage cost functions can be unbounded. In these game models, each player chooses an action each time which influences the transition probability of a Markov chain controlled only by this player. Moreover, each player needs to pay some costs which depend on the actions of all the players. First, we give an existence condition of stationary constrained Nash equilibria based on the technique of average occupation measures and the best response linear program. Then, combining the best response linear program and duality program, we present a non-convex mathematic program and prove that each stationary Nash equilibrium is a global minimizer of this mathematic program. Finally, a controlled wireless network is presented to illustrate our main results.

1. Introduction

Stochastic games introduced by Shapley in Reference [1] which have been actively pursued over the last few decades because of several applications mainly in economics and queueing system; see, for instance, References [2,3,4,5]. In this paper, we study the special constrained stochastic games with independent state processes. More precisely, each player chooses an action and pays some costs which depend on the actions of all players in each stage. However, each player’s action only influence the transition probability of the Markov chain controlled by herself. In such case, the player would wish to minimize certain expected average cost which she is most concerned about, while wants to keep other kinds of expected average costs within bounds. The game models under these formulation have been considered in References [6,7]. In Reference [6], the authors present an existence condition of stationary Nash equilibria for the constrained average games. Reference [7] derives a necessary and sufficient condition for a stationary Nash equilibria to be a global minimizer of a mathematic program.

Different from the framework of game model considered in this paper, References [2,8,9] study the so-called centralized stochastic games in which all players jointly control a single Markov chain and the one-stage costs of each player are influenced by the actions of all players. Reference [8] considers the game model with expected discounted cost criteria and expected average cost criteria and yields the existence of stationary Nash equilibria in the context of finite state space and compact action spaces. Using vanishing discount approach, Reference [9] generalizes the existence result in Reference [8] to the average games with denumerable states. Reference [2] extends the existence result in Reference [8] to the discounted game with denumerable states but under some special hypothesis on the transition laws and boundness condition imposed on the one-stage costs. By special finite-state approximation approach, Reference [10] weakens the above existence condition in Reference [8]. For applications of constrained games, Reference [11] studies the wireless powered networks including a user and an attacker. Based on the constrained game theory in Reference [6], Reference [11] gives the energy request and data transmission strategy of the user and the attack strategy of the attacker in the sense of Nash equilibria. Reference [12] considers the wireless networks in which each mobil wants to maximize its expected capacity subject to some power and buffer length constraints. Reference [13] introduces a neighbor-aware energy-efficient monitoring system for energy harvesting Internet of Things. A constrained stochastic game model is established to minimize the number of transmissions while keeping a desired monitoring probability and a best response dynamics-based algorithm is developed in Reference [13].

In this paper, we consider the constrained average games with denumerable states and unbounded costs. The main contributions of the present paper are as follows. First, by introducing the average occupation measures, we establish the so-called best-response linear program to characterize the occupation measures of stationary constrained Nash equilibria as fixed points of certain multifunction from the product space of occupation measures into itself. The standard weak convergence technique used in References [6,7] can not apply directly to the case wherein the costs are unbounded. Meanwhile, the arguments in References [6,7] employ the finiteness of the state spaces. However, the costs are always unbounded and the state spaces are not finite in some controlled stochastic models such as the stochastic inventory model, queueing system; see, for instance References [14,15]. Therefore, we introduce the so-called w-weak convergence topology and impose the standard drift condition on the transition kernel, the growth condition and additive condition on the costs, w-uniform geometrical ergodicity condition and continuity-compactness conditions. By the properties of w-weak convergence, we study the asymptotic properties of average occupation measures and expected average costs, which are used to establish the upper-continuity of the multifunction. Then, we show the existence of stationary constrained Nash equilibria by the fixed-point theorem. It should be mentioned that the vanishing discount approach in Reference [9] is based on the existence of discounted Nash equilibria in Reference [10], that is to show the limit of discounted Nash equilibria is the average Nash equilibrium when the discount factors tend to one. However, in this paper, we use fixed-point method directly because the existence of discounted Nash equilibria for stochastic games with independent state process has not been established. Finally, we characterize each stationary constrained Nash equilibrium as a global minimizer of this mathematical program which can be viewed as the combination of the best response linear program and duality program.

The rest of the paper is organized as follows. In Section 2, we introduce the constrained average game model we are concerned with. In Section 3, we introduce the best response linear program and study the corresponding convergent property of the best response linear program based on the average occupation measures. In Section 4, we establish the main statements, including the existence result and the characterization of stationary constrained Nash equilibria. In Section 5, a controlled wireless network is presented to illustrate our main results.

2. The Game Model

If S is a Borel space, we denote by $\mathcal{B}\left(S\right)$ its Borel $\sigma$-algebra, by $\mathbb{P}\left(S\right)$ the set of all probability measures on $\mathcal{B}\left(S\right)$ endowed with the topology of weak convergence, and by $\mathbb{B}\left(S\right)$ the set of all Borel measurable functions on S. Let I stand for the indicator function, $N\ge 1$ be a fixed integer. Let us define $\mathcal{I}:=\{1,\dots ,N\}$, and use indexs i or j to denote a player. Now, we introduce the N-person constrained game model

$$\mathcal{G}:=\left\{{\left(X^i,A^i,\{A^i\left(x^i\right)|x^i\in X^i\},{\left\{c_k^i\right\}}_{k=0}^p,{\left\{d_k^i\right\}}_{k=1}^p,Q^i(·|x^i,a^i),{\nu }^i\right)}_{i\in \mathcal{I}}\right\}.$$

(1)

For each $i\in \mathcal{I}$, the state space $X^i$ is assumed to be a denumerable set and $A^i$ is the action space which is assumed to be a Polish space endowed with the Borel $\sigma$-algebra $\mathcal{B}\left(A^i\right)$. Without loss of generality, we assume $X^i:=\{0,1,\dots \}$ and is enumerated by the natural order. Let $\mathit{X}:={\prod }_{i=1}^N{X}^i$ be the product space of the state spaces and $\mathit{A}:={\prod }_{i=1}^N{A}^i$ be the product space of the action spaces. For each $x^i\in X^i$, the nonempty measurable subset $A^i\left(x^i\right)$ of $A^i$ denotes the set of actions available when the state of player i is $x^i\in X^i$. Let $K^i:=\left\{(x^i,a^i)|x^i\in X^i,a^i\in A^i\left(x^i\right)\right\}$ and $K:={\prod }_{i=1}^N{K}^i$. $Q^i$ is the stochastic kernel on $X^i$ given $K^i$ controlled by player i, that is $Q^i(y^i|x^i,a^i)$ is the probability of moving from state $x^i$ to $y^i$ if player i chooses action $a^i$. For each $0\le k\le p$, the one-stage cost $c_k^i$ is a real-valued measurable function on $K^i$. The constants $d_k^i$ ($1\le k\le p$) denote the constraints and ${\nu }^i$ denotes the initial distribution of player i. Let $\nu :=({\nu }^1,\dots ,{\nu }^N)$.

For each $i\in \mathcal{I}$, let $H_0^i:=X^i,H_t^i:={\left(K^i\right)}^t\times X^i$ be the set of all histories up to time t and ${\Phi }^i$ be the set of all stochastic kernels on $A^i$ given $X^i$ for each i, where ${\left(K^i\right)}^t$ denotes the t-power product space of $K^i$.

Definition 1.

(1) 

A randomized history-dependent strategy for player i is a sequence ${\pi }^i:=\{{\pi }_t^i,$ $t=0,1,2,\dots \}$ of stochastic kernels ${\pi }_t^i$ on the action space $A^i$ given $H_t^i$ satisfying

$${\pi }_t^i(A^i\left(x_t^i\right)|h_t^i)=1\forall h_t^{i}t=0,1,2,\cdots .$$

(2) 

A randomized history-dependent strategy ${\pi }^i=\left\{{\pi }_t^i\right\}\in {\Pi }_h^i$ for player i is said to be randomized stationary if there is a stochastic kernel ${\phi }^i\in {\Phi }^i$ such that ${\pi }_t^i(·|h_t^i)={\phi }^i(·|x_t^i)$ for each $h_t^i\in H_t$. We will write such a stationary strategy as ${\phi }^i$.

Assumption 1.

The players do not observe their costs, that is, the strategy chosen by any player does not depend on the realization of the cost.

Remark 1.

Assumption 1 is imposed on [6,7], which is used to ensure that a player could not use the one-stage costs to estimate the state and the action of the other players.

The sets of all randomized history-dependent strategies and randomized stationary strategies for player i are denoted by ${\Pi }_h^i$ and ${\Pi }_s^i$, respectively. A multi-strategy is a vector $\pi :=({\pi }^1,\dots ,{\pi }^N)\in {\Pi }_h$, where ${\Pi }_h:={\times }_{i=1}^N{\Pi }_h^i$. Let ${\Pi }_s:={\times }_{i=1}^N{\Pi }_s^i$ denote the set of all randomized stationary multi-strategies.

Let ${\Omega }^i:={(X^i\times A^i)}^{\infty }$ and ${\mathcal{F}}^i$ be the corresponding product $\sigma$-algebra. Then, for each ${\pi }^i\in {\Pi }_h^i$ and each initial distribution ${\nu }^i\in \mathcal{P}\left(X^i\right)$, the well-known Tulcea’s Theorem ([16], p. 178) ensures the existence of a unique probability measure $P_{{\nu }^i}^{{\pi }^i}$ on $({\Omega }^i,{\mathcal{F}}^i)$ such that, for each $B^i\in \mathcal{B}\left(X^i\right)$ and $h_t^i\in H_t^i$,

$$\begin{array}{c}\hfill P_{{\nu }^i}^{{\pi }^i}(x_{t+1}^i\in B^i|h_t^i,a_t^i)=Q^i(B^i|x_t^i,a_t^i),t=0,1,2,\cdots ,\end{array}$$

(2)

where $x_t^i$ and $a_t^i$ denote the state and the action at the decision epoch t, respectively. The expectation operator with respect to $P_{{\nu }^i}^{{\pi }^i}$ is denoted by $E_{{\nu }^i}^{{\pi }^i}$. If ${\nu }^i$ is concentrated at some state $x^i\in X^i$, we will write $P_{{\nu }^i}^{{\pi }^i}$ and $E_{{\nu }^i}^{{\pi }^i}$ as $P_{x^i}^{{\pi }^i}$ and $E_{x^i}^{{\pi }^i}$, respectively. For each $\pi :=({\pi }^1,\dots ,{\pi }^N)\in {\Pi }_h$, the product measure of $P_{{\nu }^i}^{{\pi }^i}$ is denoted by ${\tilde{P}}_{\nu }^{\pi }:={\times }_{i=1}^N{P}_{{\nu }^i}^{{\pi }^i}$ and the corresponding expectation operator is denoted by ${\tilde{E}}_{\nu }^{\pi }$. For each ${\widehat{\pi }}^i\in {\Pi }_h^i$, we denote by $[{\pi }^{-i},{\widehat{\pi }}^i]$ the N-vector multi-strategy obtained from $\pi$ by replacing ${\pi }^i$ with ${\widehat{\pi }}^i$. Similarly, for each $a^i\in A^i$, we denote by $[{\pi }^{-i},a^i]$ the N-vector in which the jth component is ${\pi }^j$ for $j\ne i$, while the ith component is $a^i$.

For each $\nu \in {\times }_{i=1}^N\mathbb{P}\left(X^i\right)$ and $\pi \in {\Pi }_h$, the expected average criteria are defined, for each player $i\in \mathcal{I}$ and $0\le k\le p$, as

$$V_k^i(\nu ,\pi ):=\underset{n\to \infty }{lim\; sup}\frac{1}{n}{\tilde{E}}_{\nu }^{\pi }\left[\sum _{t=0}^{n-1}{c}_k^i({\mathit{x}}_t,{\mathit{a}}_t)\right],$$

where ${\mathit{x}}_t$ and ${\mathit{a}}_t$ denote the state vector and action vector at time t, respectively.

Definition 2.

(1) 

For a fixed $\pi =({\pi }^1,\dots {\pi }^N)\in {\Pi }_h$, a strategy ${\pi }^{\prime i}\in {\Pi }_h^i$ is said to be feasible for player i against π if $V_k^i(\nu ,[{\pi }^{-i},{\pi }^{\prime i}])\le d_k^i$ for each $1\le k\le p$. Let

$${\Delta }^i\left(\pi \right):=\left\{{\pi }^{\prime i}\in {\Pi }_h^i|V_k^i(\nu ,[{\pi }^{-i},{\pi }^{\prime i}])\le d_k^i,\mathit{for}\mathit{each}1\le k\le p\right\},$$

be the set of all feasible strategies for player i against π.

(2) 

$\pi :=({\pi }^1,\dots ,{\pi }^N)\in {\Pi }_h$ is called a feasible multi-strategy for $\mathcal{G}$ if ${\pi }^i$ is in ${\Delta }^i\left(\pi \right)$ for each $i\in \mathcal{I}$. We denote by Δ the set of all feasible multi-strategies.

(3) 

(Nash equilibrium) ${\pi }^{*}$ is called a constrained Nash equilibrium if ${\pi }^{*}\in \Delta$ and

$$V_0^i(\nu ,{\pi }^{*})=\underset{{\pi }^{\prime i}\in {\Delta }^i\left({\pi }^{*}\right)}{inf}{V}_0^i(\nu ,[{\pi }^{*-i},{\pi }^{\prime i}]).$$

A constrained Nash equilibrium ${\pi }^{*}$ is said to be stationary if ${\pi }^{*}$ belongs to ${\Pi }_s$.

3. The Technique Preliminary

A monotone nondecreasing function $f:X^i\to [1,\infty )$ such that ${lim}_{x^i\to \infty }f\left(x^i\right)=\infty$ will be refereed to as a Lyapunov function on $X^i$ for each $i\in \mathcal{I}$. For each function g and constant $\tau$, we denote by $g^{\tau }$ the $\tau$ power of the function g.

To guarantee the finiteness of the expected average costs, we need the following drift conditions, which is widely used in References [14,16,17,18] for discrete-time Markov decision processes and in References [5,10] for stochastic games.

Assumption 2.

For each $i\in \mathcal{I}$, there exist a Lyapunov function $w_i\ge 1$ on $X^i$, constants $b_i\ge 0,1>{\beta }_i>0$ and $\tau >1$ such that

(1) 

${\sum }_{y^i\in X^i}{w}_i^{\tau }\left(x^i\right)Q^i(y^i|x^i,a^i)\le {\beta }_i{w}_i^{\tau }\left(x^i\right)+b_i$;

(2)

${\sum }_{x^i\in X^i}{w}_i^{\tau }\left(x^i\right){\nu }^i\left(x^i\right)<\infty$.

By Jensen’s inequality, it follows that there exist constants ${\beta }_i\left({\tau }^{\prime }\right):={\beta }_i^{\frac{{\tau }^{\prime }}{\tau }}>0$ and $b_i\left({\tau }^{\prime }\right):=b_i^{\frac{{\tau }^{\prime }}{\tau }}>0$ (depending on ${\tau }^{\prime }$) for each $0<{\tau }^{\prime }<\tau$ such that

$$\sum _{y^i\in X^i}{Q}^i(y^i|x^i,a^i)w_i^{{\tau }^{\prime }}\left(y^i\right)\le {\beta }_i\left({\tau }^{\prime }\right)w_i^{{\tau }^{\prime }}\left(x^i\right)+b_i\left({\tau }^{\prime }\right),\mathrm{for}\mathrm{each}(x^i,a^i)\in K^i.$$

(3)

Now, we give two different kinds of hypotheses imposed on the cost functions to ensure the existence of Nash equilibria.

Assumption 3.

(1) 

(Growth condition) There exists a constant $M_1>0$ such that $|c_k^i(x^1,\dots ,x^N,a^1,\dots ,a^N)|\le M_1\underset{i\in \mathcal{I}}{min}\left\{w_i\left(x^i\right)\right\}$ for each $(x^1,\dots ,x^N,a^1,\dots ,a^N)\in K$ and $0\le k\le p$.

(2) 

The function $c_k^i(x^1,\dots ,x^N,a^1,\dots ,a^N)$ is continuous in $(a^1,\dots ,a^N)\in {\times }_{i=1}^N{A}^i\left(x^i\right)$ for each fixed $(x^1,\dots ,x^N)\in \mathit{X}$.

Assumption 4.

Suppose that there exist a constant $M_2>0$ and functions $f_k^{i,j}$ on $K^j$ with $j\in \mathcal{I}$ for each $i\in \mathcal{I}$ and $0\le k\le p$, such that

(1) 

$c_k^i(x^1,\dots ,x^N,a^1,\dots ,a^N)={\sum }_{j=1}^N{f}_k^{i,j}(x^j,a^j)$ and $|f_k^{i,j}(x^j,a^j)|\le M_2{w}_j\left(x^j\right)$;

(2) 

$f_k^{i,j}(x^j,·)$ is continuous in $a^j\in A^j\left(x^j\right)$ for each $x^j\in S^j$.

Remark 2.

(1) 

It should be mentioned that the cost functions in References [6,7,9] are assumed to be bounded. However, the cost functions are always unbounded in some controlled stochastic models such as the stochastic inventory model, queueing system; see, for instance, References [14,15].

(2) 

As in Assumption 4, the case that the immediate costs are additive is also considered in References [7,19] for discrete-time unconstrained stochastic games.

Lemma 1.

Suppose that Assumptions 2 and 3(1) (resp. 4(1)) hold. For each $i\in \mathcal{I}$, ${\pi }^i\in {\Pi }_h^i$ and $x^i\in X^i$,

$$\underset{n\to \infty }{lim\; sup}\frac{1}{n}{E}_{{\nu }^i}^{{\pi }^i}\left[\sum _{t=0}^{n-1}{w}_i^{\tau }\left(x_t^i\right)\right]\le \frac{b_i}{1-{\beta }_i},$$

(4)

$$V_k^i(\nu ,\pi )\le M_1\frac{b_i}{1-{\beta }_i}(\mathrm{resp}.V_k^i(\nu ,\pi )\le M_2\sum _{j=1}^N\frac{b_j}{1-{\beta }_j}).$$

(5)

Proof. 

(4) and (5) follow directly from Lemma 10.4.1 in Reference [17]. □

For each function $u\in \mathbb{B}\left(X^i\right)$ (resp. $u\in \mathbb{B}\left(K^i\right)$) and $1\le f\in \mathbb{B}\left(X^i\right)$, let us define the f-norm ${||u||}_f$ by ${||u||}_f:={sup}_{x^i\in X^i}\frac{|u(x^i)|}{f\left(x^i\right)}$ (resp. ${||u||}_f:={sup}_{x^i\in X^i}\frac{|u(x^i,a^i)|}{f\left(x^i\right)}$) and the Banach space ${\mathbb{B}}_f\left(X^i\right):=\left\{u\in B\left(X^i\right){|||u||}_f<\infty \right\}$ (resp. ${\mathbb{B}}_f\left(K^i\right):=\left\{u\in \mathbb{B}\left(K^i\right){|||u||}_f<\infty \right\}$). The set of all bounded measurable functions on $X^i$ is denoted by ${\mathbb{B}}_1\left(X^i\right)$.

For each $i\in \mathcal{I}$ and ${\phi }^i\in {\Pi }_s^i$, let us define

$$Q^i(y^i|x^i,{\phi }^i):={\int }_{A^i}{Q}^i(y^i|x^i,a^i){\phi }^i(d{a}^i|x^i).$$

For each $t\ge 1$, let $Q^i(x^i|y^i,{\phi }^i,t)$ denote the t-step transition probability from $y^i$ to $x^i$ corresponding to ${\phi }^i$. Obviously, $Q^i(x^i|y^i,{\phi }^i,1)=Q^i(y^i|x^i,{\phi }^i)$.

Definition 3.

For each $i\in \mathcal{I}$ and ${\phi }^i\in {\Pi }_s^i$, the transition kernel $Q^i(·|·,\phi )$ is said to be irreducible, if $P_{x^i}^{{\phi }^i}(x_t^i=y^i\mathit{for}\mathit{some}t\ge 1)>0$ for all $x^i,y^i\in X^i$.

Assumption 5.

For each $i\in \mathcal{I}$,

(1) 

let ${\phi }^i\in {\Pi }_s^i$, there exist a probability measure ${\mu }_{{\phi }^i}^i$ on $X^i$ and constants $R_i>0$ and ${\rho }_i>0$ such that

$$|\sum _{y^i\in X^i}u\left(y^i\right)Q^i(y^i|x^i,{\phi }^i,t)-{\mu }_{{\phi }^i}^i\left(u\right){|\le ||u||}_{w_i^{\tau }}{R}_i{\rho }_i^t{w}_i^{\tau }\left(x^i\right)$$

for each $u\in {\mathbb{B}}_{w_i^{\tau }}\left(X^i\right)$;

(2) 

$Q^i(·|·,\phi )$ is irreducible for all ${\phi }^i\in {\Pi }_s^i$.

Remark 3.

(1) 

Under Assumptions 2 and 5, it is easy to see that ${\mu }_{{\phi }^i}^i$ is the unique invariant probability measure of the transition kernel $Q^i(y^i|x^i,{\phi }^i)$, see Reference ([17], p.12). Moreover, we have ${\mu }_{{\phi }^i}^i\left(w_i^{{\tau }^{\prime }}\right):={\sum }_{y^i\in X^i}{w}_i^{{\tau }^{\prime }}\left(y^i\right){\mu }_{{\phi }^i}^i\left(y^i\right)\le \frac{b_i}{1-{\beta }_i}<\infty$ for each $0<{\tau }^{\prime }\le \tau$ and ${\mu }_{{\phi }^i}^i\left(x^i\right)>0$ for each $x^i\in X^i$.

(2) 

The w-uniform geometrical ergodicity condition in Assumption 5(1) has been widely used in discrete-time Markov decision processes, see References [17,18] and the reference therein. Since the state space in References [6,7] is finite, the standard ergodicity condition is only required in References [6,7]. Moreover, as the cost functions in Reference [9] are assumed to be bounded, the weaker uniform geometrical ergodicity condition is imposed on [9].

For each $i\in \mathcal{I}$, we define the average occupation measure ${\tilde{\mu }}_{{\phi }^i}^i(x^i,d{a}^i):={\mu }_{{\phi }^i}^i\left(x^i\right){\phi }^i(d{a}^i|x^i)$ for each ${\phi }^i\in {\Pi }_s^i$ and the set

$${\mathcal{N}}^i:=\{{\tilde{\mu }}_{{\phi }^i}^i|{\phi }^i\in {\Pi }_s^i\}\mathit{and}\mathcal{N}:={\times }_{i=1}^N{\mathcal{N}}^i.$$

For each $\tilde{\mu }=({\tilde{\mu }}_{{\phi }^1}^1,\dots ,{\tilde{\mu }}_{{\phi }^N}^N)\in {\times }_{i=1}^N{\mathcal{N}}^i$ and $i\in \mathcal{I}$, we introduce the following Markov decision processes ${\mathcal{M}}_{\tilde{\mu }}^i$:

$${\mathcal{M}}_{\tilde{\mu }}^i:=\left\{X^i,(A^i,\{A^i\left(x^i\right)|x^i\in X^i\}),Q^i(y^i|x^i,a^i),c_{0,\tilde{\mu }}^i(x^i,a^i),(c_{k,\tilde{\mu }}^i(x^i,a^i),d_k^i,1\le k\le p)\right\}$$

where

$$\begin{array}{cc}\hfill c_{k,\tilde{\mu }}^i(x^i,a^i):=& {\int }_{K^N}{\tilde{\mu }}_{{\phi }^N}^N(x^N,d{a}^N)\dots {\int }_{K^{i+1}}{\tilde{\mu }}_{{\phi }^{i+1}}^{i+1}(x^{i+1},d{a}^{i+1}){\int }_{K^{i-1}}{\tilde{\mu }}_{{\phi }^{i-1}}^{i-1}(x^{i-1},d{a}^{i-1})\hfill \\ & \dots {\int }_{K^1}{\tilde{\mu }}_{{\phi }^1}^1(x^1,d{a}^1)c_k^i(x^1,\dots ,x^N,a^1,\dots ,a^N),\hfill \end{array}$$

for each $0\le k\le p$ and $(x^i,a^i)\in K^i$, the other components are the same as in (1).

For each $i\in \mathcal{I}$ and ${\pi }^i\in {\Pi }_h^i$, the expected average cost is defined by

$$V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i):=\underset{n\to \infty }{lim\; sup}\frac{1}{n}{E}_{{\nu }^i}^{{\pi }^i}\left[\sum _{t=0}^{n-1}{c}_{k,\tilde{\mu }}^i(x_t^i,a_t^i)\right]\mathrm{for}\mathrm{each}0\le k\le p.$$

Definition 4.

For a fixed $i\in \mathcal{I}$, strategy ${\pi }^i\in {\Pi }_h^i$ is said to be feasible for ${\mathcal{M}}_{\tilde{\mu }}^i$ if

$$V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\le d_k^i\mathit{for}\mathit{each}1\le k\le p.$$

Let $U^i\left(\tilde{\mu }\right)$ be the set of all feasible strategies of ${\mathcal{M}}_{\tilde{\mu }}^i$. A strategy ${\pi }^i\in {\Pi }_h^i$ is said to be optimal for ${\mathcal{M}}_{\tilde{\mu }}^i$ if ${\pi }^i\in U^i\left(\tilde{\mu }\right)$ and

$$V_{0,\tilde{\mu }}^i({\nu }^i,{\pi }^i)=\underset{{\pi }^{\prime i}\in U^i\left(\tilde{\mu }\right)}{inf}{V}_{0,\tilde{\mu }}^i({\nu }^i,{\pi }^{\prime i}).$$

We denote by $V_{\tilde{\mu }}^{\ast i}$ the optimal value of ${\mathcal{M}}_{\tilde{\mu }}^i$.

For each $i\in \mathcal{I}$ and $\tilde{\mu }\in \mathcal{N}$, under Assumptions 2, 3 (resp. 4) and 5, we have

$$V_{k,\tilde{\mu }}^i({\nu }^i,{\phi }^i)={\int }_{K^i}{c}_{k,\tilde{\mu }}^i(x^i,a^i){\tilde{\mu }}_{{\phi }^i}^i(x^i,d{a}^i)\mathrm{for}\mathrm{each}{\phi }^i\in {\Pi }_s^i.$$

Now, we introduce the following constrained optimality problem

$$\begin{array}{cc}\hfill \mathrm{minimize}& V_{0,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\hfill \\ \hfill \mathrm{subject}\mathrm{to}& {\pi }^i\in {\Pi }_h^i,V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\le d_k^i,1\le k\le p.\hfill \end{array}$$

(6)

Lemma 2.

Suppose that Assumptions 1, 2 and 5 are satisfied and in addition either Assumption 3 or 4 holds. Let $\tilde{\mu }:=\{{\tilde{\mu }}^1,\dots ,{\tilde{\mu }}^N\}\in \mathcal{N}$ and $\phi :=({\phi }^1,\dots ,{\phi }^N)$ with ${\tilde{\mu }}^i(x^i,d{a}^i)={\widehat{\tilde{\mu }}}^i\left(x^i\right){\phi }^i(d{a}^i|x^i)$ for each $i\in \mathcal{I}$. Then

$$V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)=V_k^i(\nu ,[{\phi }^{-i},{\pi }^i])\mathit{for}\mathit{each}{\pi }^i\in {\Pi }_h^i\mathit{and}0\le k\le p.$$

Proof. 

We only prove for $i=1$. Let us define

$$\begin{array}{ccc}\hfill c_k^1(x^1,\dots ,x^N,[{\phi }^{-1},a^1])& :=& {\int }_{A^2}{\phi }^2(d{a}^2|x^2)\dots {\int }_{A^N}{c}_k^1(x^1,\dots ,x^N,\mathit{a}){\phi }^N(d{a}^N|x^N),\hfill \end{array}$$

$$p_{{\nu }^j}(t,x^j,{\phi }^j):=\sum _{y^j\in X^j}{\nu }^j\left(y^j\right)Q^j(x^j|y^j,{\phi }^j,t)\mathrm{for}\mathrm{each}{\phi }^j\in {\Pi }_s^j.$$

First, we assume Assumption 3 holds. It follows from Assumption 5 that, for each player $j\ne 1$, we have

$$\begin{array}{cc}& |\sum _{x^j\in X^j}{\tilde{\mu }}_{{\phi }^j}^j\left(x^j\right)c_k^1(x^1,\dots ,x^N,[{\phi }^{-1},a^1])-\sum _{x^j\in X^j}{p}_{{\nu }^j}(t,x^j,{\phi }^j)c_k^1(x^1,\dots ,x^N,[{\phi }^{-1},a^1])|\hfill \\ \le & {\rho }_j^t{R}_j||c_k^1(x^1,\dots ,x^{j-1},x^{j+1},\dots ,x^N,[{\phi }^{-1},a^1]){||}_{w_j^{\tau }}\sum _{x^j\in X^j}{\nu }^j\left(x^j\right)w_j^{\tau }\left(x^j\right)\hfill \\ \le & {\rho }_j^t{R}_j{M}_1\sum _{x^j\in X^j}{\nu }^j\left(x^j\right)w_j^{\tau }\left(x^j\right),\hfill \end{array}$$

where $||c_k^1(x^1,\dots ,x^{j-1},x^{j+1},\dots ,x^N,[{\phi }^{-1},a^1]){||}_{w_j^{\tau }}:=\underset{x^j\in X^j}{sup}\frac{|c_k^1(x^1,\dots ,x^N,[{\phi }^{-1},a^1])|}{w_j^{\tau }\left(x^j\right)}$ for each $0\le k\le p$. For notational ease, we define ${\prod }_{n=2}^{j-1}{\widehat{\tilde{\mu }}}^n\left(x^n\right):=1$ for $j=2$ and ${\prod }_{n=j+1}^N{p}_{{\nu }^n}(t,x^n,{\phi }^n):=1$ for $j=N$ and ${\mathit{X}}^{-1}:={\prod }_{i=2}^N{X}^i$. Hence,

$$\begin{array}{cc}& |E_{{\nu }^1}^{{\pi }^1}\left[c_{k,\tilde{\mu }}^1(x_t^1,a_t^1)\right]-{\tilde{E}}_{\nu }^{[{\phi }^{-1},{\pi }^1]}\left[c_k^1({\mathit{x}}_t,[{\phi }^{-1},a_t^1])\right]|\hfill \\ =& |{\int }_{K^1}{P}_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\sum _{x^2\in X^2}{\widehat{\tilde{\mu }}}^2\left(x^2\right)\cdots \sum _{x^N\in X^N}{\widehat{\tilde{\mu }}}^N\left(x^N\right)c_k^1(x_t^1,\dots ,x^N,[{\phi }^{-1},a_t^1])\hfill \\ & -{\int }_{K^1}{P}_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\sum _{x^2\in X^2}{p}_{{\nu }^2}(t,x^2,{\phi }^2)\cdots \sum _{x^N\in X^N}{p}_{{\nu }^N}(t,x^N,{\phi }^N)c_k^1(x_t^1,\dots ,x^N,[{\phi }^{-1},a_t^1])|\hfill \\ \le & |\sum _{j=2}^N{\int }_{K^1}{P}_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\sum _{(x^2,\dots ,x^N)\in {\mathit{X}}^{-1}}\prod _{n=2}^{j-1}{\widehat{\tilde{\mu }}}^n\left(x^n\right)·\prod _{n=j+1}^N{p}_{{\nu }^n}(t,x^n,{\phi }^n)\hfill \\ & ·({\widehat{\tilde{\mu }}}^j\left(x^j\right)-p_{{\nu }^j}(t,x^j,{\phi }^j))c_k^1(x_t^1,\dots ,x^N,[{\phi }^{-1},a_t^1])|\hfill \\ \le & \sum _{j=2}^N\left[R_j{\rho }_j^t{M}_1\sum _{x^j\in X^j}{\nu }^j\left(x^j\right)w_j^{\tau }\left(x^j\right)\right].\hfill \end{array}$$

Thus, we have that

$$\begin{array}{cc}& \frac{1}{n}\sum _{t=0}^{n-1}|E_{{\nu }^1}^{{\pi }^1}\left[c_{k,\tilde{\mu }}^1(x_t^1,a_t^1)\right]-{\tilde{E}}_{\nu }^{[{\phi }^{-1},{\pi }^1]}\left[c_k^1({\mathit{x}}_t,[{\phi }^{-1},a_t^1])\right]|\hfill \\ \le & \frac{1}{n}\sum _{j=2}^N\left(R_j{M}_1\sum _{x^j\in X^j}{\nu }^j\left(x^j\right)w_j^{\tau }\left(x^j\right)\right)·\sum _{t=0}^{n-1}{\rho }_j^t\to 0,\mathrm{as}n\to \infty ,\hfill \end{array}$$

which implies the desired result. On the other hand, if Assumption 4 holds,

$$\begin{array}{cc}& V_k^1(\nu ,[{\phi }^{-1},{\pi }^1])\hfill \\ =& \underset{n\to \infty }{lim\; sup}\frac{1}{n}\sum _{t=0}^{n-1}\left[\sum _{j=2}^N\sum _{x^j\in X^j}{f}_k^{1,j}(x^j,{\phi }^j)p_{{\nu }^j}(t,x^j,{\phi }^j)+{\int }_{K^1}{f}_k^{1,1}(x_t^1,a_t^1)P_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\right]\hfill \\ =& \sum _{j=2}^N\sum _{x^j\in X^j}{f}_k^{1,j}(x^j,{\phi }^j){\widehat{\tilde{\mu }}}^j\left(x^j\right)+\underset{n\to \infty }{lim\; sup}\frac{1}{n}\sum _{t=0}^{n-1}\left[{\int }_{K^1}{f}_k^{1,1}(x_t^1,a_t^1)P_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\right]\hfill \\ =& \underset{n\to \infty }{lim\; sup}\frac{1}{n}\sum _{t=0}^{n-1}\left[\sum _{j=2}^N\sum _{x^j\in X^j}{f}_k^{1,j}(x^j,{\phi }^j){\widehat{\tilde{\mu }}}^j\left(x^j\right)+{\int }_{K^1}{f}_k^{1,1}(x_t^1,a_t^1)P_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)\right]\hfill \\ =& \underset{n\to \infty }{lim\; sup}\frac{1}{n}\sum _{t=0}^{n-1}\left[{\int }_{K^1}{P}_{{\nu }^1}^{{\pi }^1}(x_t^1,d{a}_t^1)c_{k,\tilde{\mu }}^1(x_t^1,a_t^1)\right]=V_{k,\tilde{\mu }}^1({\nu }^1,{\pi }^1).\hfill \end{array}$$

 □

Let $w_i$ be the function as in Assumption 2 for each $i\in \mathcal{I}$. Let us define ${\mathbb{C}}_{w_i}\left(K^i\right):=\{u\in {\mathbb{B}}_{w_i}\left(K^i\right)|u\mathrm{is}\mathrm{continuous}\mathrm{on}{K}^i\}$ and ${\mathbb{P}}_{w_i}\left(K^i\right):=\{\mu \in \mathbb{P}\left(K^i\right)|{\int }_{K^i}{w}_i\left(x^i\right)\mu (x^i,d{a}^i)<\infty \}$.

Definition 5.

For each $i\in \mathcal{I}$, let $\left\{{\mu }_n\right\}\subseteq {\mathbb{P}}_{w_i}\left(K^i\right)$. ${\mu }_n$ is said to converge to $\mu \in \mathbb{P}\left(K^i\right)$ with respect to the $w_i$-weak topology if and only if ${lim}_{n\to \infty }{\int }_{K^i}g(x^i,a^i){\mu }_n(x^i,d{a}^i)={\int }_{K^i}g(x^i,a^i)\mu (x^i,d{a}^i)$ for each $g\in {\mathbb{C}}_{w_i}\left(K^i\right).$ In this case, we denote it by ${\mu }_n\stackrel{w_i}{\to }\mu$.

Let $\mathit{w}:=(w_1,\dots ,w_N)$ and $\left\{{\mu }_n\right\}\subseteq {\times }_{i=1}^N{\mathbb{P}}_{w_i}\left(K^i\right)$ with ${\mu }_n=({\mu }_n^1,\dots ,{\mu }_n^N)$, we say ${\mu }_n\stackrel{\mathit{w}}{\to }\mu =:({\mu }^1,\dots ,{\mu }^N)$ if ${\mu }_n^i\stackrel{w_i}{\to }{\mu }^i$ for each $i\in \mathcal{I}$. For each $i\in \mathcal{I}$ and ${\mu }^i\in {\mathbb{P}}_{w_i}\left(K^i\right)$, we define ${\widehat{\mu }}^i\left(x^i\right):={\mu }^i(x^i,A^i\left(x^i\right))$ for each $x^i\in X^i$.

To ensure the existence of constrained Nash equilibria, we also need the following standard continuity-compactness conditions which are widely used; see, for instance, References [4,14,16,17,19] and the reference therein.

Assumption 6.

(1) 

For each $i\in \mathcal{I}$ and $x^i\in X^i$, $A^i\left(x^i\right)$ is a compact set.

(2) 

The functions $Q^i(y^i|x^i,a^i)$ is continuous in $a^i\in A^i\left(x^i\right)$ for each fixed $i\in \mathcal{I}$ and $x^i,y^i\in X^i$.

(3) 

The functions ${\sum }_{y^i\in X^i}{w}_i\left(y^i\right)Q^i(y^i|x^i,a^i)$ is continuous in $a^i\in A^i\left(x^i\right)$ for each fixed $i\in \mathcal{I}$ and $x^i\in X^i$.

Lemma 3.

Suppose that Assumptions 2, 5 and 6 hold. For each $i\in \mathcal{I}$, the set ${\mathcal{N}}^i$ is convex and compact with respect to the $w_i$-weak topology.

Proof. 

Fix $1\le {\tau }^{\prime }<\tau$, it follows from Remark 3 that $\underset{{\tilde{\mu }}^i\in {\mathcal{N}}^i}{sup}\left\{{\int }_{K^i}{w}_i^{\tau }\left(x^i\right){\tilde{\mu }}^i(x^i,d{a}^i)\right\}\le \frac{b_i}{1-{\beta }_i}$ and the set $\{(x^i,a^i)\in K^i|w_i^{\tau }\left(x^i\right)\le n{w}_i^{{\tau }^{\prime }}\left(x^i\right)\}$ is compact in $K^i$ for each $n\ge 1$. Hence, it follows from Corollary A.46 in Reference [20], the set ${\mathcal{N}}^i$ is compact with respect to the $w_i$-weak topology. The other statement can be obtained by Lemma 5.2.2 in Reference [18]. □

Remark 4.

The standard weak convergence technique used in References [6,7] for bounded costs does not apply directly to the case wherein costs are unbounded in this paper.

Lemma 4.

Suppose that Assumptions 1, 2, 5 and 6 are satisfied and in addition either Assumption 3 or 4 holds. Let $i\in \mathcal{I}$ be fixed, $\left\{{\tilde{\mu }}_n\right\}\subseteq \mathcal{N}$ and $\left\{{\eta }_n\right\}\subseteq {\mathcal{N}}^i$, such that ${\tilde{\mu }}_n\stackrel{\mathit{w}}{\to }{\tilde{\mu }}_{\infty }$ and ${\eta }_n\stackrel{w_i}{\to }{\eta }_{\infty }$ weakly in $\mathbb{P}\left(K^i\right)$. Then,

$$\underset{n\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\eta }_n(x^i,d{a}^i)={\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i)$$

for each $0\le k\le p$.

Proof. 

For each $n\in \overline{\mathbb{N}}$, we assume ${\tilde{\mu }}_n:=({\tilde{\mu }}_n^1,\dots ,{\tilde{\mu }}_n^N)$. According to proposition D.8 in Reference [16], there exist ${\phi }_n^i$ and ${\tilde{\phi }}_n^i\in {\Pi }_s^i$ such that ${\tilde{\mu }}_n^i(x^i,d{a}^i)={\widehat{\tilde{\mu }}}_n^i\left(x^i\right){\phi }_n^i(d{a}^i|x^i)$ and ${\eta }_n(x^i,d{a}^i)={\widehat{\eta }}_n\left(x^i\right){\tilde{\phi }}_n^i(d{a}^i|x^i)$ for each $i\in \mathcal{I}$. For an arbitrary $u\in {\mathbb{B}}_1\left(X^i\right)$, it follows from Lemma 5.1.2 in Reference [18] that

$${\int }_{K^i}\left(\sum _{y^i\in X^i}u\left(y^i\right)Q^i(y^i|x^i,a^i)-u\left(x^i\right)\right){\tilde{\mu }}_n^i(x^i,d{a}^i)=0.$$

Since ${\tilde{\mu }}_n^i\stackrel{w_i}{\to }{\tilde{\mu }}_{\infty }^i$, it follows from Assumption 6(2) that

$$\begin{array}{cc}& \underset{n\to \infty }{lim}{\int }_{K^i}\left(\sum _{y^i\in X^i}u\left(y^i\right)Q^i(y^i|x^i,a^i)-u\left(x^i\right)\right){\tilde{\mu }}_n^i(x^i,d{a}^i)\hfill \\ =& {\int }_{K^i}\left(\sum _{y^i\in X^i}u\left(y^i\right)Q^i(y^i|x^i,a^i)-u\left(x^i\right)\right){\tilde{\mu }}_{\infty }^i(x^i,d{a}^i)=0,\hfill \end{array}$$

which together with Lemma 5.1.2 in Reference [18] implies that ${\tilde{\mu }}_{\infty }^i\in {\mathcal{N}}^i$ for each $i\in \mathcal{I}$. Similarly, we also have ${\eta }_{\infty }\in {\mathcal{N}}^i$. Thus, we can see that

$$\underset{n\to \infty }{lim}{\widehat{\tilde{\mu }}}_n^i\left(x^i\right)={\widehat{\tilde{\mu }}}_{\infty }^i\left(x^i\right)>0\mathit{and}\underset{n\to \infty }{lim}{\widehat{\eta }}_n\left(x^i\right)={\widehat{\eta }}_{\infty }\left(x^i\right)>0,\mathrm{for}\mathrm{each}{x}^i\in X^i,$$

(7)

${\tilde{\phi }}_n^i(·|x^i)\to {\tilde{\phi }}_{\infty }^i(·|x^i)$ and ${\phi }_n^i(·|x^i)\to {\phi }_{\infty }^i(·|x^i)$ weakly in $\mathbb{P}\left(A^i\left(x^i\right)\right)$ for each $x^i\in X^i$. It follows from Assumption 3(2) or Assumption 4 that

$$\underset{n\to \infty }{lim}{c}_k^i(x^1,\dots ,x^N,[{\phi }_n^{-i},{\tilde{\phi }}_n^i])=c_k^i(x^1,\dots ,x^N,[{\phi }_{\infty }^{-i},{\tilde{\phi }}_{\infty }^i]),$$

(8)

for each $(x^1,\dots ,x^N)\in \mathit{X}$, where ${\phi }_n:=({\phi }_n^1,\dots ,{\phi }_n^N)$ for each $n\in \overline{\mathbb{N}}$.

Under Assumptions 2(2) and 5, we have that

$$\sum _{x^1\in X^1}{\widehat{\tilde{\mu }}}_{\infty }^1\left(x^1\right)w_1\left(x^1\right)<\infty \mathit{and}\underset{n\to \infty }{lim}\sum _{x^i\in X^i}{w}_i\left(x^i\right){\widehat{\eta }}_n\left(x^i\right)=\sum _{x^i\in X^i}{w}_i\left(x^i\right){\widehat{\eta }}_{\infty }\left(x^i\right)\forall i\in \mathcal{I}.$$

(9)

Then, under Assumption 3(2), by (7)–(9) and Proposition A.2.6 in Reference [15], it follows that

$$\begin{array}{cc}& \underset{n\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\eta }_n(x^i,d{a}^i)\hfill \\ =& \underset{n\to \infty }{lim}\sum _{x^i\in X^i}{\widehat{\eta }}_n\left(x^i\right){\int }_{A^i\left(x^i\right)}{\tilde{\phi }}_n^i(d{a}^i|x^i)\sum _{x^N\in X^N}{\widehat{\tilde{\mu }}}_n^N\left(x^N\right){\int }_{A^N\left(x^N\right)}{\phi }_n^N(d{a}^N|x^N)\dots \sum _{x^{i+1}\in X^{i+1}}{\widehat{\tilde{\mu }}}_n^{i+1}\left(x^{i+1}\right)\hfill \\ & {\int }_{A^{i+1}\left(x^{i+1}\right)}{\phi }_n^{i+1}(d{a}^{i+1}|x^{i+1})\sum _{x^{i-1}\in X^{i-1}}{\widehat{\tilde{\mu }}}_n^{i-1}\left(x^{i-1}\right){\int }_{A^{i-1}\left(x^{i-1}\right)}{\phi }_n^{i-1}(d{a}^{i-1}|x^{i-1})\hfill \\ & \dots {\int }_{A^1\left(x^1\right)}{\phi }_n^1(d{a}^1|x^1)\sum _{x^1\in X^1}{\widehat{\tilde{\mu }}}_n^1\left(x^1\right)c_k^i(x^1,\dots ,x^N,a^1,\dots ,a^N)\hfill \\ =& {\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i).\hfill \end{array}$$

If Assumption 3 is replaced by Assumption 4, then we have

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\eta }_n(x^i,d{a}^i)& =& \underset{n\to \infty }{lim}[{\int }_{K^i}{f}_k^{i,i}(x^i,a^i){\eta }_n(x^i,d{a}^i)+\sum _{j=1,j\ne i}^N{\int }_{K^j}{f}_k^{i,j}(x^j,a^j){\tilde{\mu }}_n^j(x^j,d{a}^j)]\hfill \\ & =& {\int }_{K^i}{f}_k^{i,i}(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i)+\sum _{j=1,j\ne i}^N{\int }_{K^j}{f}_k^{i,j}(x^j,a^j){\tilde{\mu }}_{\infty }^j(x^j,d{a}^j)\hfill \\ & =& {\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i).\hfill \end{array}$$

 □

The following slater condition is common for constrained games, see References [2,6,7,8,9,10].

Assumption 7.

(Slater condition) For each stationary multi-strategy $\phi \in {\Pi }_s$ and each player i, there exists ${\tilde{\pi }}^i\in {\Pi }_h^i$ such that

$$V_k^i(\nu ,[{\phi }^{-i},{\tilde{\pi }}^i])<d_k^i,\mathit{for}\mathit{each}1\le k\le p.$$

Lemma 5.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Let $i\in \mathcal{I}$ and $\tilde{\mu }:=({\tilde{\mu }}^1,\dots ,{\tilde{\mu }}^N)\in \mathcal{N}$ be fixed. Then,

(1) 

for each ${\pi }^i\in {\Pi }_h^i$, there exists a stationary strategy ${\overline{\phi }}^i\in {\Pi }_s^i$ (depending on $\tilde{\mu }$, ${\nu }^i$ and ${\pi }^i$), such that $V_{k,\tilde{\mu }}^i({\nu }^i,{\overline{\phi }}^i)\le V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)$ for each $0\le k\le p$;

(2) 

there exists a stationary strategy ${\tilde{\phi }}^i\in {\Pi }_s^i$ such that $V_{k,\tilde{\mu }}^i({\nu }^i,{\tilde{\phi }}^i)<d_k^i$ for each $1\le k\le p$.

Proof. 

(1)

See Lemma 5.7.10 in Reference [16].

(2)

Let $\phi :=({\phi }^1,\dots ,{\phi }^N)\in {\Pi }_s$ such that ${\tilde{\mu }}^i(x^i,d{a}^i)={\mu }_{{\phi }^i}^i\left(x^i\right){\phi }^i(d{a}^i|x^i)$ for each $i\in \mathcal{I}$. For the fixed $\phi$, let ${\tilde{\pi }}^i\in {\Pi }_h^i$ be the corresponding strategy as in Assumption 7. It follows from Lemma 2 that $V_{k,\tilde{\mu }}^i({\nu }^i,{\tilde{\pi }}^i)=V_k^i(\nu ,[{\phi }^{-i},{\tilde{\pi }}^i])<d_k^i$ for each $1\le k\le p$. By part (1), there exists a stationary strategy ${\tilde{\phi }}^i\in {\Pi }_s^i$ (depending on $\phi$ and ${\tilde{\pi }}^i$) such that $V_{k,\tilde{\mu }}^i({\nu }^i,{\tilde{\phi }}^i)\le V_{k,\tilde{\mu }}^i({\nu }^i,{\tilde{\pi }}^i)=V_k^i(\nu ,[{\phi }^{-i},{\tilde{\pi }}^i])<d_k^i$ for each $1\le k\le p$.

 □

It follows from Lemma 5.1.2 in Reference [18] that $\eta \in {\mathcal{N}}^i$ if and only if

$$\sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}(Q^i(y^i|x^i,a^i)-I_{\left\{y^i\right\}}\left(x^i\right))\eta (x^i,d{a}^i)=0,\mathrm{for}\mathrm{each}i\in \mathcal{I}.$$

According to Lemma 5, the constrained optimality problem (6) can be restricted to the set of all stationary strategies. Hence, for each $\tilde{\mu }\in \mathcal{N}$ and $i\in \mathcal{I}$, the constrained optimality problem (6) is equivalent to the following best response linear program $\left({\mathrm{LP}}_{\tilde{\mu }}^i\right)$:

$$\begin{array}{cc}\hfill {\mathrm{LP}}_{\tilde{\mu }}^i:& \underset{\eta \in {\mathbb{P}}_{w_i}\left(K^i\right)}{inf}{\int }_{K^i}{c}_{0,\tilde{\mu }}^i(x^i,a^i)\eta (x^i,d{a}^i)\hfill \\ & \mathrm{subject}\mathrm{to}\left\{\begin{array}{c}{\int }_{K^i}{c}_{k,\tilde{\mu }}^i(x^i,a^i)\eta (x^i,d{a}^i)\le d_k^i,1\le k\le p,\hfill \\ {\int }_{K^i}(Q^i(y^i|x^i,a^i)-I_{\left\{y^i\right\}}\left(x^i\right))\eta (x^i,d{a}^i)=0\mathrm{for}\mathrm{each}{y}^i\in X^i.\hfill \end{array}\right.\hfill \end{array}$$

(10)

For a fixed $i\in \mathcal{I}$ and $\tilde{\mu }\in \mathcal{N}$, $\eta$ is said to be feasible for LP${}_{\tilde{\mu }}^i$ if it satisfies (10). The sets of all feasible solutions and optimal solutions are denoted by ${\mathit{F}}^i\left(\tilde{\mu }\right)$ and ${\mathit{O}}^i\left(\tilde{\mu }\right)$, respectively. The optimal value of ${\mathrm{LP}}_{\tilde{\mu }}^i$ is denoted by $inf{\mathrm{LP}}_{\tilde{\mu }}^i$.

Proposition 1.

Suppose that Assumptions 1, 2, 5, 6 and 7 and are satisfied and in addition either Assumption 3 or 4 holds. Let $\tilde{\mu }:=({\tilde{\mu }}^1,\dots ,{\tilde{\mu }}^N)\in \mathcal{N}$ and $\phi :=({\phi }^1,\dots ,{\phi }^N)\in {\Pi }_s$ such that ${\tilde{\mu }}^i(x^i,d{a}^i)={\widehat{\tilde{\mu }}}^i\left(x^i\right){\phi }^i(d{a}^i|x^i)$ for each $i\in \mathcal{I}$. Then,

(1) 

${\phi }^i$ is an optimal strategy of ${\mathcal{M}}_{\tilde{\mu }}^i$ for each $i\in \mathcal{I}$ if and only if φ is a stationary constrained Nash equilibrium;

(2) 

${\tilde{\mu }}^i$ is an optimal strategy of LP${}_{\tilde{\mu }}^i$ for each $i\in \mathcal{I}$ if and only if φ is a stationary constrained Nash equilibrium.

Proof. 

(1)

⟹. Let $i\in \mathcal{I}$ be fixed and ${\pi }^i\in {\Delta }^i\left(\phi \right)$. It follows from Lemma 2 that

$$V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)=V_k^i(\nu ,[{\phi }^{-i},{\pi }^i])\le d_k^i\mathit{for}\mathit{each}1\le k\le p.$$

Thus, we have ${\pi }^i\in U^i\left(\tilde{\mu }\right)$. Then, Lemma 2 yields that

$$V_0^i(\nu ,[{\phi }^{-i},{\pi }^i])=V_{0,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\ge V_{0,\tilde{\mu }}^i({\nu }^i,{\phi }^i)=V_0^i(\nu ,\phi ),$$

which implies that $\phi$ is a constrained Nash equilibrium.

⟸. For each $i\in \mathcal{I}$, we take an arbitrary strategy ${\pi }^i\in {\Pi }_h^i$ such that $V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\le d_k^i$ for each $1\le k\le p$. Lemma 2 yields that $V_k^i(\nu ,[{\phi }^{-i},{\pi }^i])=V_{k,\tilde{\mu }}^i({\nu }^i,{\pi }^i)\le d_k^i$ for each $1\le k\le p$, which implies that ${\pi }^i\in {\Delta }^i\left(\phi \right)$. As $\phi$ is a constrained Nash equilibrium, it follows that $V_{0,\tilde{\mu }}^i({\nu }^i,{\phi }^i)=V_0^i(\nu ,\phi )\le V_0^i(\nu ,[{\phi }^{-i},{\pi }^i])\le V_{0,\tilde{\mu }}^i({\nu }^i,{\pi }^i)$, which implies that ${\phi }^i$ is an optimal strategy of ${\mathcal{M}}_{\tilde{\mu }}^i$.

(2)

${\tilde{\mu }}^i$ is optimal for LP${}_{\tilde{\mu }}^i$ if and only if ${\phi }^i$ is optimal for ${\mathcal{M}}_{\tilde{\mu }}^i$. Hence, we get the desired result by part (1). □

Lemma 6.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Let $i\in \mathcal{I}$ be fixed and $\left\{{\tilde{\mu }}_n\right\}\subseteq \mathcal{N}$ such that ${\tilde{\mu }}_n\stackrel{\mathit{w}}{\to }{\tilde{\mu }}_{\infty }$, and ${\eta }_n\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right)$ for each $n\in \mathbb{N}$. Then, $\left\{{\eta }_n\right\}$ is relatively compact in ${\mathcal{N}}^i$ with respect to the $w_i$-weak topology and the accumulation point is a feasible solution of LP${}_{{\tilde{\mu }}_{\infty }}^i$.

Proof. 

It follows from Lemma 3 that there exists a subsequence $\left\{{\eta }_{n_m}\right\}$ such that ${\eta }_{n_m}\stackrel{w_i}{\to }{\eta }_{\infty }$ with respect to the $w_i$-weak topology. Then, Lemma 4 yields that

$${\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i)=\underset{m\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_{n_m}}^i(x^i,a^i){\eta }_{n_m}(x^i,d{a}^i)\le d_k^i.$$

(11)

As $Q^i(·|x^i,a^i)\in {\mathbb{C}}_{w_i}\left(K^i\right)$, it follows that

$${\int }_{K^i}(Q^i(y^i|x^i,a^i)-I_{\left\{y^i\right\}}\left(x^i\right)){\eta }_{\infty }(x^i,d{a}^i)=\underset{m\to \infty }{lim}{\int }_{K^i}(Q^i(y^i|x^i,a^i)-I_{\left\{y^i\right\}}\left(x^i\right)){\eta }_{n_m}(x^i,d{a}^i)=0,$$

which together with (11) implies that ${\eta }_{\infty }\in {\mathit{F}}^i\left({\tilde{\mu }}_{\infty }\right)$. □

Lemma 7.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Let $i\in \mathcal{I}$ be fixed and take an arbitrary $\tilde{\mu }\in \mathcal{N}$, then LP${}_{\tilde{\mu }}^i$ has an optimal solution.

Proof. 

First, it follows from Lemma 5(2) that ${\mathit{F}}^i\left(\tilde{\mu }\right)\ne \varnothing$. Let $\left\{{\eta }_m\right\}\subseteq {\mathit{F}}^i\left(\tilde{\mu }\right)$ be the minimizing sequence such that

$$\underset{m\to \infty }{lim}\sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}{c}_{0,\tilde{\mu }}^i(x^i,a^i){\eta }_m(x^i,d{a}^i)=inf{\mathrm{LP}}_{\tilde{\mu }}^i.$$

(12)

By Lemma 6, there exists a subsequence $\left\{{\eta }_{m_s}\right\}$ of $\left\{{\eta }_m\right\}$ such that ${\eta }_{m_s}\stackrel{w_i}{\to }{\eta }_{\infty }\in {\mathit{F}}^i\left(\tilde{\mu }\right)$. Then, it follows from Lemma 4 and (12) that

$$\sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}{c}_{0,\tilde{\mu }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i)=\underset{s\to \infty }{lim}\sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}{c}_{0,\tilde{\mu }}^i(x^i,a^i){\eta }_{m_s}(x^i,d{a}^i)=inf{\mathrm{LP}}_{\tilde{\mu }}^i.$$

This is ${\eta }_{\infty }\in {\mathit{O}}^i\left(\tilde{\mu }\right)$. □

The idea of the proof of the following lemma is from Lemma 5.1 and Theorem 3.9 in Reference [21] for constrained discrete-time Markov decision processes with discounted cost criteria.

Lemma 8.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Let $i\in \mathcal{I}$ be fixed and $\left\{{\tilde{\mu }}_n\right\}\subseteq \mathcal{N}$ such that ${\tilde{\mu }}_n\stackrel{\mathit{w}}{\to }{\tilde{\mu }}_{\infty }$. Then,

(1) 

for each $\eta \in {\mathit{F}}^i\left({\tilde{\mu }}_{\infty }\right)$, there exist an integer N and ${\eta }_n\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right)$ for all $n\ge N$, such that ${\eta }_n\stackrel{w_i}{\to }\eta$ weakly in $\mathbb{P}\left(K^i\right)$;

(2) 

if ${\eta }_{\infty }^{\prime }$ is an accumulation point of sequence $\{{\eta }_n^{\prime },n\ge 1\}$ in which ${\eta }_n^{\prime }$ is an optimal solution of LP${}_{{\tilde{\mu }}_n}^i$ for each $n\in \mathbb{N}$, then ${\eta }_{\infty }^{\prime }\in {\mathit{O}}^i\left({\tilde{\mu }}_{\infty }\right).$

Proof. 

(1)

Lemma 5(2) gives the existence of ${\phi }^i,{\tilde{\phi }}^i\in {\Pi }_s^i$ and constant $D>0$, such that $\eta ={\tilde{\mu }}_{{\phi }^i}^i$ and $V_{k,{\tilde{\mu }}_{\infty }}^i({\nu }^i,{\tilde{\phi }}^i)\le d_k^i-D$ for all $1\le k\le p$. By Lemma 4, it follows that

$$\begin{array}{c}\underset{n\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i)\eta (x^i,d{a}^i)={\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i)\eta (x^i,d{a}^i)\le d_k^i,\hfill \end{array}$$

(13)

$$\begin{array}{c}\underset{n\to \infty }{lim}{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\tilde{\mu }}_{{\tilde{\phi }}^i}^i(x^i,d{a}^i)={\int }_{K^i}{c}_{k,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\tilde{\mu }}_{{\tilde{\phi }}^i}^i(x^i,d{a}^i)\le d_k^i-D,\hfill \end{array}$$

(14)

for all $1\le k\le p.$

Let $\epsilon$ be such that $0<\epsilon <\frac{D}{2}$. By (13)-(14), there exists an integer $N_{\epsilon }$ (depending on $\epsilon$) such that, for each $n\ge N_{\epsilon }$ and $1\le k\le p$,

$${\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i)\eta (x^i,d{a}^i)\le d_k^i+\epsilon ,{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\tilde{\mu }}_{{\tilde{\phi }}^i}^i(x^i,d{a}^i)\le d_k^i-D+\epsilon .$$

Let ${\nu }_n^{\epsilon }:=(1-{\lambda }_{\epsilon })\eta +{\lambda }_{\epsilon }{\tilde{\mu }}_{{\tilde{\phi }}^i}^i,$ where ${\lambda }_{\epsilon }:=\frac{\epsilon }{D}<\frac{1}{2}$. Then, we derive from Lemma 3 that ${\nu }_n^{\epsilon }\in {\mathcal{N}}^i$, and

$$\begin{array}{ccc}\hfill {\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\nu }_n^{\epsilon }(x^i,d{a}^i)& =& (1-{\lambda }_{\epsilon }){\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i)\eta (x^i,d{a}^i)+{\lambda }_{\epsilon }{\int }_{K^i}{c}_{k,{\tilde{\mu }}_n}^i(x^i,a^i){\tilde{\mu }}_{{\tilde{\phi }}^i}^i(x^i,d{a}^i)\hfill \\ \hfill & \le & (1-{\lambda }_{\epsilon })(d_k^i+\epsilon )+{\lambda }_{\epsilon }[(d_k^i-D)+\epsilon ]\le d_k^i,\hfill \end{array}$$

(15)

for each $1\le k\le p$ and $n\ge N_{\epsilon }$, which implies that ${\nu }_n^{\epsilon }\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right)$ for all $n\ge N_{\epsilon }$.

Then, let $\left\{{\epsilon }_s\right\}\subseteq \mathbb{R}$, such that ${\epsilon }_s\downarrow 0$ and $0<{\epsilon }_s<\frac{D}{2}$. For each fixed $s\ge 1$ (corresponding to a given ${\epsilon }_s$), as in the previous argument, there exists an integer $N_s$ (depending on s) which is assumed to be increasing in $s\ge 1$, such that

$${\nu }_n^s:=(1-{\lambda }_s)\eta +{\lambda }_s{\tilde{\mu }}_{{\tilde{\phi }}^i}^i\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right),\forall n\ge N_s,s\ge 1,\mathrm{where}{\lambda }_s:=\frac{{\epsilon }_s}{D}.$$

(16)

Let

$${\eta }_n:={\nu }_n^s\mathit{and}{\lambda }_n:={\lambda }_s,\mathrm{for}\mathrm{each}{N}_s\le n<N_{s+1}.$$

(17)

Since ${\epsilon }_s\downarrow 0$, by (15)–(17), we have ${\eta }_n\stackrel{w_i}{\to }\eta$ weakly in $\mathbb{P}\left(K^i\right)$ as $n\to \infty$ and ${\eta }_n\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right)$ for each $n\ge N_1$, which completes the proof of part (i).

(2)

By Lemma 6, without loss of generality, we assume that ${\eta }_n^{\prime }\stackrel{w_i}{\to }{\eta }_{\infty }^{\prime }\in {\mathit{F}}^i\left({\tilde{\mu }}_{\infty }\right)$ weakly in $\mathbb{P}\left(K^i\right)$. On the other hand, for any ${\eta }_{\infty }\in {\mathit{F}}^i\left({\tilde{\mu }}_{\infty }\right)$, it follows from part (1) that there exist an integer N and ${\eta }_n\in {\mathit{F}}^i\left({\tilde{\mu }}_n\right)$ for all $n\ge N$, such that ${\eta }_n\stackrel{w_i}{\to }{\eta }_{\infty }$ weakly in $\mathbb{P}\left(K^i\right)$, which together with ${\eta }_n^{\prime }\in {\mathit{O}}^i\left({\tilde{\mu }}_n\right)$, gives

$${\int }_{K^i}{c}_{0,{\tilde{\mu }}_n}^i(x^i,a^i){\eta }_n^{\prime }(x^i,d{a}^i)\le {\int }_{K^i}{c}_{0,{\tilde{\mu }}_n}^i(x^i,a^i){\eta }_n(x^i,d{a}^i)\forall n\ge N.$$

(18)

Then, by Lemma 4 and (18), we have

$${\int }_{K^i}{c}_{0,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }^{\prime }(x^i,d{a}^i)\le {\int }_{K^i}{c}_{0,{\tilde{\mu }}_{\infty }}^i(x^i,a^i){\eta }_{\infty }(x^i,d{a}^i),$$

which implies ${\eta }_{\infty }^{\prime }\in {\mathit{O}}^i\left({\tilde{\mu }}_{\infty }\right)$.

4. The Main Results

In this section, we give the existence and characterization of stationary constrained Nash equilibria. For each $\tilde{\mu }=({\tilde{\mu }}^1,\dots ,{\tilde{\mu }}^N)\in \mathcal{N}$, we introduce the following two multi-functions:

$${\Lambda }^i:\mathcal{N}\to 2^{{\mathcal{N}}^i}\mathrm{and}\Psi \left(\tilde{\mu }\right):\mathcal{N}\to 2^{\mathcal{N}}\mathrm{for}\mathrm{each}i\in \mathcal{I}$$

$${\Lambda }^i\left(\tilde{\mu }\right):=\{\eta \in {\mathcal{N}}^i|\eta \mathrm{is}\mathrm{an}\mathrm{optimal}\mathrm{solution}\mathrm{of}{\mathrm{LP}}_{\tilde{\mu }}^i\}\mathrm{and}\Psi \left(\tilde{\mu }\right):={\times }_{i=1}^N{\Lambda }^i\left(\tilde{\mu }\right).$$

(19)

Theorem 1.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Then, there exists a stationary constrained Nash equilibrium.

Proof. 

Let $i\in \mathcal{I}$ be fixed, $\left\{{\tilde{\mu }}_n\right\}\subseteq \mathcal{N}$ and ${\eta }_n\in {\Lambda }^i\left({\tilde{\mu }}_n\right)$ such that ${\tilde{\mu }}_n\stackrel{\mathit{w}}{\to }{\tilde{\mu }}_{\infty }$ and ${\eta }_n\stackrel{w_i}{\to }{\eta }_{\infty }$. Then, by Lemma 8(2), we know that ${\eta }_{\infty }\in {\mathit{O}}^i\left({\tilde{\mu }}_{\infty }\right)$ which implies that ${\Lambda }^i$ is upper semi-continuous. By the arbitrariness of i, we can derive that $\Psi$ is upper semi-continuous. Moreover, Lemmas 3, 7 and (10) deduce that the set $\Psi \left(\tilde{\mu }\right)$ is nonempty and convex for each $\tilde{\mu }\in \mathcal{N}$. Moreover, we can see that $\Psi \left(\tilde{\mu }\right)$ is compact by the upper semi-continuity of $\Psi$. Hence, it follows from Fan’s fixed point Theorem in Reference [22] that that $\Psi$ has a fixed point ${\tilde{\mu }}^{*}$. By Proposition D.8 in Reference [16], there exists ${\phi }^{*}:=({\phi }^{*1},\dots ,{\phi }^{*N})\in {\Pi }_s$ such that ${\mu }^{*i}(x^i,d{a}^i)={\widehat{\mu }}^{*i}\left(x^i\right){\phi }^{*i}(d{a}^i|x^i)$, which together with Proposition 1, implies that ${\phi }^{*}$ is a stationary constrained Nash equilibrium. □

By the same proof of Theorems 5.3.1 and 5.4.1 in Reference [18] for discrete-time Markov decision processes, we can yield the following statements.

Lemma 9.

Suppose that Assumptions 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Then, for each given $\tilde{\mu }$,

(1) 

there exists a function $h^{*i}\in {\mathbb{B}}_{{\omega }_i}\left(X^i\right)$ and a vector ${\lambda }^{*i}:=({\lambda }_1^{*i},\dots ,{\lambda }_p^{*i})\in {[0,\infty )}^p$ for each $i\in \mathcal{I}$ such that

$$\begin{array}{ccc}\hfill V_{\tilde{\mu }}^{*i}+h^{*i}\left(x^i\right)& =& \underset{a^i\in A^i\left(x^i\right)}{inf}\{c_{0,\tilde{\mu }}^i(x^i,a^i)+\sum _{k=1}^p{\lambda }_k^{*i}(c_{k,\tilde{\mu }}^i(x^i,a^i)-d_k^i)+\sum _{y^i\in X^i}{h}^{*i}\left(y^i\right)Q^i(y^i|x^i,a^i)\};\hfill \end{array}$$

(2) 

for each ${\lambda }^i=({\lambda }_1^i,\dots ,{\lambda }_p^i)\in {[0,\infty )}^p$, there exists $(V_{{\lambda }^i}^i,h^i)\in \mathbb{R}\times {\mathbb{B}}_{{\omega }_i}\left(X^i\right)$ such that

$$V_{{\lambda }^i}^i+h^i\left(x^i\right)=\underset{a^i\in A^i\left(x^i\right)}{inf}\{c_{0,\tilde{\mu }}^i(x^i,a^i)+\sum _{k=1}^p{\lambda }_k^i(c_{k,\tilde{\mu }}^i(x^i,a^i)-d_k^i)+\sum _{y^i\in X^i}{h}^i\left(y^i\right)Q^i(y^i|x^i,a^i)\},$$

and $V_{\tilde{\mu }}^{*i}={inf}_{\lambda \in {[0,\infty )}^p}{V}_{{\lambda }^i}^i$.

Now we introduce the following duality program ${\mathrm{DP}}_{\tilde{\mu }}^i$ for each $\tilde{\mu }\in \mathcal{N}$ and $i\in \mathcal{I}$:

$$\begin{array}{cc}\hfill {\mathrm{DP}}_{\tilde{\mu }}^i:& \underset{(h^i,{\lambda }^i)\in {\mathbb{B}}_{w_i}\left(X^i\right)\times {[0,\infty )}^p}{sup}{V}_{{\lambda }^i}^i\hfill \\ & \mathrm{subject}\mathrm{to}\left\{\begin{array}{c}{V}_{{\lambda }^i}^i+h^i\left(x^i\right)\le c_{0,\tilde{\mu }}^i(x^i,a^i)+\sum _{k=1}^p{\lambda }_k^i(c_{k,\tilde{\mu }}^i(x^i,a^i)-d_k^i)\hfill \\ +\sum _{y^i\in X^i}{h}^i\left(y^i\right)Q^i(y^i|x^i,a^i)\mathrm{for}\mathrm{each}(x^i,a^i)\in (X^i\times A^i),\hfill \\ {\lambda }_k^i\ge 0\mathrm{for}\mathrm{each}1\le k\le p.\hfill \end{array}\right.\hfill \end{array}$$

(20)

Combining ${\mathrm{LP}}_{\tilde{\mu }}^i$ and ${\mathrm{DP}}_{\tilde{\mu }}^i$ together, we introduce the following mathematical program $\left(\mathrm{MP}\right)$:

$$\begin{array}{cc}\hfill \mathrm{MP}:& \underset{\zeta \in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}}{min}\sum _{i=1}^N[{\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a})\prod _{i=1}^N{\tilde{\mu }}^i(x^i,d{a}^i)-V^i]\hfill \\ & \mathrm{subject}\mathrm{to}\left\{\begin{array}{c}{V}^i+h^i\left(x^i\right)\le c_{0,\tilde{\mu }}^i(x^i,a^i)+\sum _{k=1}^p{\lambda }_k^i(c_{k,\tilde{\mu }}^i(x^i,a^i)-d_k^i)\hfill \\ +\sum _{y^i\in X^i}{h}^i\left(y^i\right)Q^i(y^i|x^i,a^i)\mathrm{for}\mathrm{each}(x^i,a^i)\in (X^i\times A^i),\hfill \\ {\lambda }_k^i\ge 0\mathrm{for}\mathrm{each}1\le k\le p,\hfill \\ \sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}{c}_{k,\tilde{\mu }}^i(x^i,a^i){\tilde{\mu }}^i(x^i,d{a}^i)\le d_k^i,1\le k\le p,\hfill \\ \sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}(Q^i(y^i|x^i,a^i)-I_{\left\{y^i\right\}}\left(x^i\right)){\tilde{\mu }}^i(x^i,d{a}^i)=0\mathrm{for}\mathrm{each}{y}^i\in X^i,\hfill \end{array}\right.\hfill \end{array}$$

(21)

where $\zeta :={(V^i,h^i,{\tilde{\mu }}^i,{\lambda }^i)}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$. Let us define the function $\Phi \left(\zeta \right):={\sum }_{i=1}^N[{\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a}){\prod }_{i=1}^N{\tilde{\mu }}^i(x^i,d{a}^i)-V^i]$, for each $\zeta ={(V^i,h^i,{\tilde{\mu }}^i,{\lambda }^i)}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$.

In Reference [7], the authors characterize each stationary constrained Nash equilibrium in the finite-state games as a global minimum of certain mathematical program. The following theorem generalizes the above result to the denumerable-state games.

Theorem 2.

Suppose that Assumptions 1, 2, 5, 6 and 7 are satisfied and in addition either Assumption 3 or 4 holds. Then,

(1) 

there exists ${\zeta }^{*}={(V^{*i},h^{*i},{\tilde{\mu }}^{*i},{\lambda }^{*i})}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$ such that it is a global minimum of the mathematical program MP;

(2) 

suppose that ${\zeta }^{=}{(V^i,h^i,{\tilde{\mu }}^i,{\lambda }^i)}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$ is a global minimum of the mathematical program MP with $\Phi \left(\zeta \right)=0$. Then, there exits ${\phi }^{*i}\in {\Pi }_s^i$ such that ${\tilde{\mu }}^i(x^i,d{a}^i)={\widehat{\tilde{\mu }}}^i\left(x^i\right){\phi }^{*i}(d{a}^i|x^i)$ and ${\phi }^{*}:=({\phi }^{*1},\dots ,{\phi }^{*N})$ is a constrained Nash equilibrium of the model $\mathcal{G}$.

Proof. 

(1)

Let ${\phi }^{*}:=({\phi }^{*1},\dots ,{\phi }^{*N})$ be a constrained Nash equilibrium and let ${\tilde{\mu }}^{*}=({\tilde{\mu }}_{{\phi }^{*1}}^1,\dots ,{\tilde{\mu }}_{{\phi }^{*N}}^N)\in {\times }_{i=1}^N{\mathcal{N}}^i$ be the average occupation measure corresponding to ${\phi }^{*}$. By Proposition 1, ${\tilde{\mu }}_{{\phi }^{*i}}^i$ satisfies (10) if $\tilde{\mu }$ is replaced by ${\tilde{\mu }}^{*}$. Theorem 5.3.1 in Reference [18] and Theorem 10.3.6 in Reference [17] yield that there exists $(V^{*i},h^{*i},{\lambda }^{*i})\in \mathbb{R}\times {\mathbb{B}}_{w_i}\left(X^i\right)\times {[0,\infty )}^p$ which satisfies (20) by replacing $\tilde{\mu }$ with ${\tilde{\mu }}^{*}$ and

$$\sum _{x^i\in X^i}{\int }_{A^i\left(x^i\right)}{c}_{0,{\tilde{\mu }}^{*}}^i(x^i,a^i){\tilde{\mu }}^{*i}(x^i,d{a}^i)=V^{*i}=V_{{\tilde{\mu }}^{*}}^{*i}$$

for each $i\in \mathcal{I}$. Let ${\zeta }^{*}:={(V^{*i},h^{*i},{\tilde{\mu }}^{*i},{\lambda }^{*i})}_{i=1}^N$, we have $\Phi \left({\zeta }^{*}\right)=0$. In turn, let $\zeta ={(V_{{\lambda }^i}^i,h^i,{\tilde{\mu }}^i,{\lambda }^i)}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$ be a feasible point of $\mathrm{MP}$, we get that

$$\begin{array}{ccc}\hfill V_{{\lambda }^i}^i& \le & {\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a})\prod _{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j)+\sum _{k=1}^p{\lambda }_k^i{\int }_{\mathit{X}\times \mathit{A}}(c_k^i(\mathit{x},\mathit{a})-d_k^i)\prod _{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j)\hfill \\ & & +{\int }_{K^i}\sum _{y^i\in X^i}{h}^i\left(y^i\right)Q^i(y^i|x^i,a^i){\tilde{\mu }}^i(x^i,d{a}^i)-{\int }_{K^i}{h}^i\left(x^i\right){\tilde{\mu }}^i(x^i,d{a}^i)\hfill \\ & \le & {\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a})\prod _{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j),\hfill \end{array}$$

(22)

which implies $\Phi \left(\zeta \right)\ge 0$.

(2)

Let us take an arbitrary global minimum ${\zeta }^{=}{(V_{{\lambda }^i}^i,h^i,{\tilde{\mu }}^i,{\lambda }^i)}_{i=1}^N\in {\mathbb{R}}^N\times {\prod }_{i=1}^N({\mathbb{B}}_{w_i}\left(X^i\right)\times {\mathbb{P}}_{w_i}\left(K^i\right))\times {[0,\infty )}^{Np}$ of MP with $\Phi \left(\zeta \right)=0$. Since $\zeta$ is a feasible solution of MP, by (22), we have

$$V_{{\lambda }^i}^i\le {\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a})\prod _{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j)$$

and ${\tilde{\mu }}^i$ satisfies (10). Hence, by $\Phi \left(\zeta \right)=0$, it follows that $V_{{\lambda }^i}^i={\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a}){\prod }_{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j)$ for each $i\in \mathcal{I}$.

Then, let $\tilde{\mu }:=({\mu }^1,\dots ,{\mu }^N)$ and take an arbitrary feasible solution ${\tilde{\mu }}^{\prime i}$ of ${\mathrm{LP}}_{\tilde{\mu }}^i$, we have

$${\int }_{\mathit{X}\times \mathit{A}}(c_k^i(\mathit{x},\mathit{a})-d_k^i){\tilde{\mu }}^{\prime i}(x^i,d{a}^i)\prod _{j=1,j\ne i}^N{\tilde{\mu }}^j(x^j,d{a}^j)\le 0,\mathrm{for}\mathrm{each}1\le k\le p.$$

Then, proceeding as in the proof of (22), it follows that

$$\begin{array}{ccc}\hfill {\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a})\prod _{j=1}^N{\tilde{\mu }}^j(x^j,d{a}^j)& =& V_{{\lambda }^i}^i\le {\int }_{\mathit{X}\times \mathit{A}}{c}_0^i(\mathit{x},\mathit{a}){\tilde{\mu }}^{\prime i}(x^i,d{a}^i)\prod _{j=1,j\ne i}^N{\tilde{\mu }}^j(x^j,d{a}^j),\hfill \end{array}$$

which implies ${\tilde{\mu }}^i$ is ${\mathrm{LP}}_{\tilde{\mu }}^i$ optimal for each $i\in \mathcal{I}$. Then, Proposition D.8 in Reference [16] yields that there exists a stationary strategy ${\phi }^{*i}\in {\Pi }_s^i$ such that ${\tilde{\mu }}^i(x^i,d{a}^i)={\widehat{\tilde{\mu }}}^i\left(x^i\right){\phi }^{*i}(d{a}^i|x^i)$ for each $i\in \mathcal{I}$. By Proposition 1, ${\phi }^{*}:=({\phi }^{*1},\dots ,{\phi }^{*N})$ is a stationary constrained Nash equilibrium of the model $\mathcal{G}$.

 □

5. An Example

In this section, we use a wireless network to illustrate our conditions and main results.

Example 1.

Consider a wireless network in which there are N nodes and each node contains a mobile, a buffer and a channel. Let $x_t^i\in X^i:=\{0,1,\dots \}$ denote the number of packets in the buffer of node i and we assume that the new packets are not admitted if the buffer is not empty. At time t, if $x_t^i>0$, each mobile transmits a packet with power $a_t^i\in A^i:=[\delta ,1]$, where $0<\delta <1$. We assume each mobile will retransmit packet at time $t+1$ if the packet has not been transmitted successfully. When $x_t^i=0$, the number of the new arrivals at time t is denoted by $z_t$, which is assume to be identically distributed. The number of packets in the buffer i and the action $a_t^i$ are only available for mobile i itself. Hence, the transition probability are defined as follows: if $x^i\ge 1$

$$\begin{array}{ccc}\hfill Q^i(x^i-1|x^i,a^i)& =& a^i\hfill \\ \hfill Q^i(x^i|x^i,a^i)& =& 1-a^i,\hfill \end{array}$$

and $Q^i(x^i|0,a^i)=q\left(x^i\right)$, where ${\sum }_{k=0}^{\infty }q\left(k\right)=1$ with $q\left(k\right)>0$ for each $k\in X^i$. Let ${\nu }^i$ denote the initial distribution of the buffer i, $p_0$ be some base value of the power, $c_1^i(x^i,a^i):=p_0{a}^i$ be the cost of power for node i and $c_2^i(x^i,a^i):=x^i$ denote the delay cost for node i. As in Reference [12], we assume that the signal to interference ratio of mobile i denoted by $SI{R}_i$ is given by

$$\begin{array}{ccc}\hfill {\mathrm{SIR}}_i((x^1,x^2,\dots ,x^N),(a^1,a^2,\dots ,a^N))& :=& \frac{g_i{a}^i}{N_0+{\sum }_{j\ne i,x^j\ge 0}{r}_{ij}{g}_j{a}^j}\mathrm{if}{x}^j>0\hfill \\ & =& 0,\mathit{otherwise},\hfill \end{array}$$

where $r_{ij}$ are the coding orthogonality coefficients and $N_0$ is the thermal noise in the medium, $g_i>0$ are some constants. Let $c_0^i(x^1,\dots ,x^N,a^1,\dots ,a^N):=-{log}_2(1+{\mathrm{SIR}}_i((x^1,x^2,\dots ,x^N),(a^1,a^2,\dots ,a^N)))$ denotes the cost that each mobile wants to minimize, and $d_1^i,d_2^i$ denote the constraints of node i corresponding to $c_1^i$ and $c_2^i$.

Condition 1.

(1) 

${\sum }_{k=1}^{\infty }{k}^{2}q\left(k\right)<\infty .$

(2) 

There exists an interval $[0,\overline{s}]$ such that $m\left(s\right):={\sum }_{k=0}^{\infty }{e}^{2sk}q\left(k\right)<\infty$ for each $s\in [0,\overline{s}]$.

(3) 

There exists $\widehat{s}\in (0,\overline{s})$ such that ${\sum }_{x^i\in X^i}{e}^{2\widehat{s}{x}^i}{\nu }^i\left(x^i\right)<\infty$ for each i.

(4) 

There exists an action $a^{\prime }\in [\delta ,1]$ such that $p_0{a}^{\prime }<d_1^i$ and $\frac{1}{2}\frac{{\sum }_{k=1}^{\infty }k(1+k)q\left(k\right)}{a^{\prime }+{\sum }_{k=1}^{\infty }kq\left(k\right)}<d_2^i$ for each i.

Proposition 2.

Under condition 1, Example 1 satisfies Assumptions 1, 2, 3, 5, 6 and 7. Hence, (by Theorem 1), there exists a stationary constrained Nash equilibrium.

Proof. 

For each i and ${\phi }^i\in {\Pi }_s^i$, let ${\mu }_{{\phi }^i}^i$ denote the invariant measure and ${\phi }^i\left(x^i\right):={\int }_{\delta }^1{a}^i{\phi }^i(d{a}^i|x^i)$, which satisfies

$$\begin{array}{ccc}\hfill {\mu }_{{\phi }^i}^i\left(y^i\right)& =& \sum _{x^i=0}^{\infty }{Q}^i(y^i|x^i,{\phi }^i){\mu }_{{\phi }^i}^i\left(x^i\right)\hfill \\ & =& [1-{\phi }^i\left(y^i\right)]{\mu }_{{\phi }^i}^i\left(y^i\right)+{\phi }^i(y^i+1){\mu }_{{\phi }^i}^i(y^i+1)+q\left(y^i\right){\mu }_{{\phi }^i}^i\left(0\right).\hfill \end{array}$$

Hence, it follows from Condition 1(1) that

$$\begin{array}{ccc}\hfill {\phi }^i\left(y^i\right){\mu }_{{\phi }^i}^i\left(y^i\right)& =& {\mu }_{{\phi }^i}^i\left(0\right)q\left(y^i\right)+{\mu }_{{\phi }^i}^i\left(0\right)q(y^i+1)+{\mu }_{{\phi }^i}^i(y^i+2)q(y^i+2)\hfill \\ & =& {\mu }_{{\phi }^i}^i\left(0\right)\sum _{k=0}^{\infty }q(y^i+k),\hfill \end{array}$$

which implies that

$${\mu }_{{\phi }^i}^i\left(y^i\right)={\mu }_{{\phi }^i}^i\left(0\right)\sum _{k=0}^{\infty }\frac{q(y^i+k)}{{\phi }^i\left(y^i\right)},\mathrm{and}{\mu }_{{\phi }^i}^i\left(0\right)=\frac{1}{1+{\sum }_{x^i=1}^{\infty }{\sum }_{y^i=1}^{x^i}\frac{q\left(x^i\right)}{\phi \left(y^i\right)}}.$$

(23)

Let us define ${\beta }_1:=1-\delta (1-\frac{1}{e})$, the functions $l\left(x^i\right):=I_0\left(x^i\right)$ and $w_i\left(x^i\right):=e^{\widehat{s}{x}^i}$ and $\mu \left(x^i\right):=q\left(x^i\right)$ for each $x^i\in X^i$. It is obvious that

$$\begin{array}{ccc}\hfill Q^i(x^i-1|x^i,{\phi }^i)& =& {\phi }^i\left(x^i\right)\ge I_0\left(x^i\right)\mu (x^i-1),\hfill \\ \hfill Q^i(x^i|x^i,{\phi }^i)& =& 1-{\phi }^i\left(x^i\right)\ge I_0\left(x^i\right)\mu \left(x^i\right),\hfill \\ \hfill Q^i(y^i|0,{\phi }^i)& =& q\left(y^i\right)\ge I_0\left(0\right)q\left(y^i\right).\hfill \end{array}$$

If $x^i\ge 1$,

$$\begin{array}{ccc}\hfill \sum _{y^i\in X^i}{w}_i\left(y^i\right)Q^i(y^i|x^i,a^i)& =& e^{\widehat{s}(x^i-1)}{a}^i+e^{\widehat{s}{x}^i}(1-a^i)\hfill \\ & =& e^{\widehat{s}{x}^i}(1-a^i+\frac{1}{e}{a}^i)\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill \sum _{y^i\in X^i}{w}_i\left(y^i\right)Q^i(y^i|0,a^i)& =& \sum _{y^i\in X^i}{e}^{\widehat{s}{y}^i}q\left(y^i\right).\hfill \end{array}$$

Thus, by Condition 1(2), it follows that

$$\sum _{y^i\in X^i}{w}_i\left(y^i\right)Q^i(y^i|x^i,a^i)\le {\beta }_1{w}_i\left(x^i\right)+l\left(0\right)\sum _{y^i\in X^i}{e}^{\widehat{s}{y}^i}q\left(y^i\right).$$

Hence, the model satisfies Assumption 6(3). Let ${\beta }_2:=1-\delta (1-\frac{1}{e^{2\widehat{s}}})$ and $b_2={\sum }_{y^i\in X^i}{e}^{2\widehat{s}{y}^i}q\left(y^i\right)$. Similarly, if $x^i>0$,

$$\begin{array}{ccc}\hfill \sum _{y^i\in X^i}{w}_i^2\left(y^i\right)Q^i(y^i|x^i,a^i)& =& e^{2\widehat{s}(x^i-1)}{a}^i+e^{2\widehat{s}{x}^i}(1-a^i)\hfill \\ & =& e^{2\widehat{s}{x}^i}(1-a^i+\frac{a^i}{e^{2\widehat{s}}})\hfill \\ & \le & e^{2\widehat{s}{x}^i}{\beta }_2,\hfill \end{array}$$

if $x^i=0$,

$$\begin{array}{ccc}\hfill \sum _{y^i\in X^i}{w}_i^2\left(y^i\right)Q^i(y^i|0,a^i)& =& \sum _{y^i\in X^i}{e}^{2\widehat{s}{y}^i}q\left(y^i\right).\hfill \end{array}$$

Therefore, we have

$$\begin{array}{ccc}\hfill \sum _{y^i\in X^i}{w}_i^2\left(y^i\right)Q(y^i|x^i,a^i)& \le & {\beta }_2{w}_i^2\left(x^i\right)+b_{2}l\left(0\right),\hfill \end{array}$$

for each $x^i\ge 0$, which together with Condition 1(3) and Proposition 10.2.5 in Reference [17], implies Assumptions 2 and 5 with $\tau =2$. Let ${\phi }^{\prime i}(a^{\prime }|x^i):=1$ for each i and $\phi \in {\Pi }_s$, it follows from (23) and Condition 1(1) that

$$\begin{array}{ccc}\hfill V_2^i(\nu ,[{\phi }^{-i},{\phi }^{\prime i}])& :=& \underset{n\to \infty }{lim\; sup}\frac{1}{n}{\tilde{E}}_{\nu }^{[{\phi }^{-i},{\phi }^{\prime i}]}\left[\sum _{t=0}^{n-1}{c}_2^i(x_t^i,a_t^i)\right]\hfill \\ & =& \sum _{y^i=1}^{\infty }{y}^i{\mu }_{{\phi }^{\prime i}}^i\left(y^i\right)\hfill \\ & =& {\mu }_{{\phi }^{\prime i}}^i\left(0\right)\sum _{y^i=1}^{\infty }\sum _{k=0}^{\infty }{y}^i\frac{q(y^i+k)}{{\phi }^{\prime i}\left(y^i\right)}\hfill \\ & =& {\mu }_{{\phi }^{\prime i}}^i\left(0\right)\sum _{x^i=1}^{\infty }q\left(x^i\right)\sum _{y^i=1}^{x^i}\frac{y^i}{{\phi }^{\prime i}\left(y^i\right)}\hfill \\ & =& {\mu }_{{\phi }^{\prime i}}^i\left(0\right)\sum _{x^i=1}^{\infty }q\left(x^i\right)\frac{x^i(1+x^i)}{2a^{\prime }}\hfill \\ & =& \frac{1}{2}\frac{{\sum }_{k=1}^{\infty }k(1+k)q\left(k\right)}{a^{\prime }+{\sum }_{k=1}^{\infty }kq\left(k\right)}<d_2^i,\hfill \end{array}$$

which together with Condition 1(4) implies the Assumption 7. It is obvious that Assumptions 3 and 6(1)–(2) hold for this model. Thus, by Theorem 1, there exists a stationary constrained Nash equilibrium. □

6. Conclusions

In the present paper, we have studied the discrete-time constrained stochastic games with denumerable state space under the average cost criteria. By introducing the average occupation measures, we have established the best-response linear program and characterized the average occupation measures of stationary Nash equilibria as fixed points of ceratin multifunction. By introducing the so-called w-weak convergence topology, we have considered the asymptotic properties of average occupation measures and given the new existence condition of stationary constrained Nash equilibria. Furthermore, we have established a mathematical program and showed that each stationary Nash equilibria is a global minimizer of this mathematical program. It should be mentioned that the arguments in References [6,7] employ the compactness of the space of occupation measures with respect to the standard weak convergence topology and the solvability of best response linear program which can not apply directly for the case of unbounded costs and denumerable state space. However, the costs are usually unbounded and state space is not finite in some controlled stochastic models. Moreover, the fixed point method in the present paper is also different from the vanishing discount method in Reference [9] which is based on the existence of constrained discounted Nash equilibria. Finally, a wireless network has been presented to illustrate the applications of our main results.