Reinforcement Learning for Mean-Field Game

Abstract

Stochastic games provide a framework for interactions among multiple agents and enable a myriad of applications. In these games, agents decide on actions simultaneously. After taking an action, the state of every agent updates to the next state, and each agent receives a reward. However, finding an equilibrium (if exists) in this game is often difficult when the number of agents becomes large. This paper focuses on finding a mean-field equilibrium (MFE) in an action-coupled stochastic game setting in an episodic framework. It is assumed that an agent can approximate the impact of the other agents’ by the empirical distribution of the mean of the actions. All agents know the action distribution and employ lower-myopic best response dynamics to choose the optimal oblivious strategy. This paper proposes a posterior sampling-based approach for reinforcement learning in the mean-field game, where each agent samples a transition probability from the previous transitions. We show that the policy and action distributions converge to the optimal oblivious strategy and the limiting distribution, respectively, which constitute an MFE.

1. Introduction

1.1. Motivation

We live in a world where multiple agents interact repeatedly in a common environment. For example, multiple robots interact to achieve a specific goal. Multi-agent reinforcement learning (MARL) refers to the problem of learning and planning in a sequential decision-making system with unknown underlying system dynamics. The agents need to learn the system dynamics by trying different actions and observing rewards received over time. Learning in a MARL is fundamentally different from the traditional single-agent reinforcement learning (RL) problem since agents not only interact with the environment but also with each other. Thus, an agent, when trying to learn the underlying system dynamics, has to consider the action taken by the other agents. Changes in the policy (or actions) of any agent affect the others and vice versa.

One natural learning algorithm is to extend existing RL algorithms to the MARL by assuming that the other agents’ actions are independent. However, studies show that a smart agent which learns the joint actions of the others performs better as compared to the agent that does not learn the joint action of other agents [1,2]. For any agent, the actions of other agents become a part of the state. This results in the state space increases exponentially as the number of agents increases. When the agents are strategic, i.e., they only want to take actions that maximize their utility (or value), Nash equilibrium is often employed as the equilibrium concept. The existing equilibrium solving approaches work for some restricted games when there exists an adversarial equilibrium or coordination equilibrium [3]. Also, these approaches can handle a handful of agents because of the exponential increase in the state space. The computational complexity of finding Nash Equilibrium at every stage game prevents applications of these approaches in games where the number of agents is large [4].

In this paper, we consider MARL as an environment where a large number of agents co-exist. Similar to [5], we utilize a mean-field approach where we assume that the Q-function of an agent is affected by the mean actions of the others. Mean-field game drastically reduces the complexity, since an agent only needs to consider the empirical distribution of the actions played by other agents. Such mean-field games exist in several domains. For example, the mean-field game is observed in a cyber-security game where a large number of agents such as terminal nodes or servers make individual decisions about their security [6,7]. However, the ultimate security depends on the decisions made by other agents as well. For example, consider a network of computers, where there are a large number of agents and each agent manages a computer. If an agent invests heavily in building firewalls, its computer can still be breached if other agents’ computers are not secure. In the security game, each agent invests a certain amount to attain a security level. However, the investment level depends on the investment made by the other agents. If the number of agents is large, the game can be modeled as the mean-field game as the average investment made per agent impacts the decision of an agent.

Another example of a mean-field game is the demand response price in the smart grid [8,9]. The utility company sets a price based on the average demand per household. Hence, if at a certain time the average demand is high, the utility company may increase the price. The agent now might want to reduce its own consumption to decrease its costs resulting from the increased price. Mean-field equilibrium is the equilibrium concept in the mean-field game.

1.2. Contribution

We seek to obtain a model-based RL algorithm to find the mean-field equilibrium in an episodic-set up. To the best of our knowledge, this is the first work on an episodic RL algorithm for mean-field equilibrium. We consider an oblivious strategy [10,11], where each agent takes an action based only on its state. Thus, even though the transition probability and reward depend on the empirical distribution of the agents’ actions and states, an agent only seeks policy based on its own state. Hence, an agent does not need to track the policy state of the other agents.

In our algorithm, we maintain a history of the samples and sample an MDP that fits the history with high confidence at the start of the episode. The policy is then computed with this MDP. The agents then play the computed policy and collect samples over the steps of the episode. We show that such an algorithm converges to equilibrium.

1.3. Related Literature

Unlike the standard literature on the mean-field equilibrium on stochastic games [10,12,13], we consider that the transition probabilities are unknown to the agents. Instead, each agent learns the underlying transition probability matrix using a readily implementable posterior sampling approach [14,15]. All agents employ the best response dynamics to choose the best response strategy which maximizes the (discounted) payoff for the remaining episode length. We show the asymptotic convergence of the policy and the action distribution to the optimal oblivious strategy and the limiting action distribution respectively. We estimate the value function using backward induction and show that the value function estimates converge to the optimal value function of the true distribution. We also use the compactness of state and action space to show that the converged point constitutes a mean-field equilibrium (MFE).

Ref. [5] considers a variant of the Mean-field game where the state is the same across the agents. Unlike [5], we consider a generalized version of the game where the state can be different for different agents. Further, we do not consider a game where adversarial equilibrium or coordinated equilibrium is required to be present. We also do not need to track the action and the realized Q-value of other agents as was the case in [5].

Recently, the authors of [16] studied a policy-gradient based approach to achieve mean-field equilibrium. The authors of [17,18] considered variants of Q-learning to achieve Nash equilibrium. Actor-critic based algorithms have been analyzed in [19]. The authors of [20] consider a deep neural network-based algorithm for mean-field games. In contrast, this paper considers a posterior sampling-based approach. As noted in [21], model-based approaches converge faster than the model-free approaches in general. Thus, understanding of theoretical properties of the model-based approaches is an important problem. Further, all these papers require huge storage space as the policy depends on the actions of the other agents whereas in our setting we provide a policy that is oblivious of the states and actions of other agents.

The authors of [22,23] compute mean-field equilibrium for a setting where the evolution of the state of an agent does not depend on the action or states of the other agents. However, in our setting, we consider a generic setting where the both reward and the next state of an agent depend on the actions as well as the states of the other agents. Thus, though the mean-field game converges to the potential game in the above papers, in our setting, the game does not converge to the potential game. Hence, finding an equilibrium strategy is more challenging in our setting. The key contribution of the paper is a Posterior-sampling based algorithm that is used by each agent in a multi-agent setting, which is shown to converge to a mean-field equilibrium. The proposed algorithm does not assume the knowledge of transition probabilities and learns them using a posterior sampling approach. Further, our algorithm is computationally efficient as the policy only depends on the current state of an agent, i.e., it is oblivious of the states and actions of other agents.

2. Background

2.1. Multi-Player Stochastic Game

An n-player stochastic game is formalized by the system dynamics tuple ${\mathcal{M}}^{\ast }$ = $\{\mathcal{S},\mathcal{A},P,r,\tau ,\rho ,\gamma \}$. The agents are indexed by the set $\left[n\right]=\{1,2,\cdots ,n\}$. The state of the ith agent at time t is given by $s_{i,t}\in \mathcal{S}$, where $\mathcal{S}$ is the state space set. $\mathcal{A}\left(s\right)$ is defined as the set of the feasible actions any agent can take in state s. $\mathcal{A}$ is the action space set defined as ${\bigcup }_{s\in \mathcal{S}}\mathcal{A}\left(s\right)$. We also assume that both $\mathcal{S}$ and $\mathcal{A}$ are finite sets. Since finite sets are also compact, the assumption allows us to use results from previous works of [10]. Since we have n agents in our system, the combined state space of the system becomes ${\mathcal{S}}^{\times n}=\mathcal{S}\times \mathcal{S}\times \cdots \times \mathcal{S}$, and the combined action space of the system becomes ${\mathcal{A}}^{\times n}=\mathcal{A}\times \mathcal{A}\times \cdots \times \mathcal{A}$. Let ${\mathbf{s}}_t=\mathbf{s}\in {\mathcal{S}}^{\times n}$ be a vector of length n, and the ith element of $\mathbf{s}$ denotes the states of the ith agent at time t. Similarly, let ${\mathbf{a}}_t=\mathbf{a}\in {\mathcal{A}}^{\times n}$ be a vector of length n, and the ith element of $\mathbf{s}$ denotes the action taken by the ith agent at time t.

If the agents play joint action ${\mathbf{a}}_t=\mathbf{a}\in {\mathcal{A}}^{\times n}$ the next state of the system ${\mathbf{s}}_{t+1}={\mathbf{s}}^{\prime }\in {\mathcal{S}}^{\times n}$ follows the probability distribution $P({\mathbf{s}}_{t+1}={\mathbf{s}}^{\prime }|{\mathbf{s}}_t=\mathbf{s},{\mathbf{a}}_t=\mathbf{a})$. Along with the state updates, ith agent also receives a reward $r_{i,t}=R_i({\mathbf{s}}_t,{\mathbf{a}}_t,{\mathbf{s}}_{t+1})\in [0,1]$. We further assume that the reward function $R_i(·)$ independent of the agent i, and each player is trying to optimize for the same reward. Hence, we can drop the subscript i in the reward function $R_i$. However, since each agent can be in different state, or play different action, their individual rewards will be different and we still use the subscript i to differentiate between the instantaneous reward of the ith agent. The constant $\gamma \in [0,1)$ is the discount factor, and $\rho$ is the initial state distribution such that ${\mathbf{s}}_0\sim \rho$.

We consider an episodic framework where the length of the time horizon or the length of episodes is $\tau$. State space set $\mathcal{S}$, action space set $\mathcal{A}$, $\tau$ are known and need not be learned by the agent. We consider that the game is played in episodes $k=0,1,2,\cdots$. In each episode, the game is played in discrete steps, $j\in \left[\tau \right]=\{0,1,\cdots ,\tau -1\}$. The episodes begin at times $t_k=k\tau ,k=0,1,2,\cdots$. At each time t, the state of the agent i is given by $s_{i,t}$, the agent selects an action $a_{i,t}$, agent observes a scalar reward $r_{i,t}$ and the state transitions to the state $s_{t+1}$. Let $H_{i,t}=(s_{i,1},a_{i,1},r_{i,1},\cdots ,s_{i,t-1},a_{i,t-1},r_{i,t-1},s_{i,t})$ denote the history available to the agent i till time t.

2.2. Mean-Field Game

In a game with a large number of players, we might expect that the distribution of agents over the action space carries more meaning than the actions themselves. It is intuitive that a single agent has a negligible effect on the game as the number of agents increases. The effect of other agents on a single agent’s payoff is only via the action distribution of the population. This intuition is formalized in the mean-field game. We now formally define the mean-field game and equilibrium concepts.

First, we define few notations. Let ${\alpha }_{-i,t}\left(a\right):\mathcal{A}\to [0,1]$ be the fraction of the agents (excluding agent i) that take action $a\in \mathcal{A}$ at time t. Mathematically, we have

$$\begin{array}{c}\hfill {\alpha }_{-i,t}\left(a\right)={\displaystyle \frac{1}{n-1}}\sum _{m\in \left[n\right]\setminus \left\{i\right\}}\mathrm{𝟙}(a_{m,t}=a),\end{array}$$

(1)

where $\mathrm{𝟙}(a_{j,t}=a)$ is the indicator function that the agent j takes action a at time t. Since we assume an episodic framework, ${\alpha }_{-i,t}\left(a\right)$ can be different at each time index in an episode and also across the episodes. The episodic nature of the problem will be used later (in (9)) to define convergence to a value that depends only the time index in the episode. Further, since each agent selects exactly one action from $\mathcal{A}$, we have

$${\alpha }_{-i,t}\left(a\right)\ge 0 \forall a\in \mathcal{A}, \mathrm{and} \sum _{a\in \mathcal{A}}{\alpha }_{-i,t}\left(a\right)=1$$

Similar to distribution of agents over actions, we define $f_{-i,t}:\mathcal{S}\to [0,1]$ as the distribution of agents (excluding agent i) over the state space $\mathcal{S}$.

$$\begin{array}{c}\hfill f_{-i,t}\left(a\right)={\displaystyle \frac{1}{n-1}}\sum _{m\in \left[n\right]\setminus \left\{i\right\}}\mathrm{𝟙}(s_{m,t}=s),\end{array}$$

(2)

In a mean-field game, every agent $i\in \left[n\right]$ assumes that its next state $s_{i,t+1}$ is randomly distributed according to the transition probability distribution $P_i$ conditioned on agent’s current state $s_{i,t}$, the action taken $a_{i,t}$ and other agents’ distribution over actions ${\alpha }_{-i,t}$. Also, the reward is function of the agents current state and action and the next state.

$$s_{i,t+1}\sim P_i(·|s_{i,t},a_{i,t},{\alpha }_{-i,t})$$

(3)

$$r_{i,t}=\varphi (s_{i,t},a_{i,t},{\alpha }_{-i,t},s_{i,t+1})$$

(4)

Thus the agent does not need to concern itself with the actions of the other agents, as the population action distribution ${\alpha }_{-i,t}$ becomes a part of the environment. This updated environment dynamics can now be used in decision-making. Note that the distribution of the population action ${\alpha }_{-i,t}$ could be explicitly taken into account for deciding an action as well. Note that the reward may also depend on f the state distribution of other agents. The analysis would have been similar.

Example 1.

We now provide an example drawn from a real-life application that simulates our setting. Suppose we consider the scenario of malware spreading. The state $X_{i,t}=0$ means that an agent is vaccinated and can not infect at t. On the other hand, $X_{i,t}$ can vary between 0 and 1 with n quantization levels. The action space of an agent is $a_{i,t}=\{0,1\}$. If $a_{i,t}=0$ then the agent does not take any action. If $a_{i,t}=1$, agent i takes action in order to protect itself. In order to simplify the model, we consider the following state evolution model

$$X_{i,t+1}=\left\{\begin{array}{c}\min \{X_{i,t}+{\omega }_{i,t}{\alpha }_{-i,t}\left(0\right),1\}\hspace{1em}if a_{i,t}=0\hfill \\ 0\hspace{1em}if a_{i,t}=1\hfill \end{array}\right.$$

Note that if all the other agents have taken action 1 i.e., they protect themselves, then an agent has less chance to be infected. Thus, the state is smaller when the fraction of agents taking action 0 (i.e., ${\alpha }_{-i,t}\left(0\right)$) is smaller. ${\omega }_{i,t}$ is a noise who has its value in $\{1/n,2/n,\dots ,1\}$.

The reward function for agent i is defined as follows $r_{i,t}=-X_{i,t}{\alpha }_{-i,t}\left(1\right)-\lambda a_{i,t}$. The reward function increases if the number of agents who have protected themselves is higher. λ is the cost that depends on the action. This example explains the necessity to compute the mean-field equilibrium.

Note that in general action of an agent should depend on the action distribution of other agents. However, Proposition 1 from [10] says that under equilibrium, an oblivious strategy performs as well as a strategy that considers other agents’ actions. Thus, the strategy does not need to explicitly consider the value of ${\alpha }_{-i,t}$.

Definition 1.

An agent $i\in \left[n\right]$ is said to follow an oblivious deterministic strategy ${\pi }_i$ when the agent selects an action considering only time index j in an episode and current state $s_{i,j}$.

$$\begin{array}{cc}\hfill {\pi }_i& :\mathcal{S}\times \left[\tau \right]\to \mathcal{A}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill a_{i,j}& ={\pi }_i(s_{i,j},j)\hfill \end{array}$$

(6)

For the rest of the paper, we will focus on oblivious deterministic strategy for all agents.

2.3. Value Function, Q Function and Policy

We now define a value function for an agent $i\in \left[n\right]$ for oblivious policy ${\pi }_i$ at lth time step in an episode as:

$$\begin{array}{cc}\hfill V_{i,{\pi }_i,l}\left(s\right|{\alpha }_{-i,l})& ={\mathbf{E}}_{P_i,{\pi }_i}\left[\sum _{j=l}^{\tau -1}{\gamma }^{j-l}{r}_{i,t}|s_{i,l}=s\right]\hfill \end{array}$$

(7)

$$\begin{array}{c}={\mathbf{E}}_{P_i,{\pi }_i}\left[\sum _{j=l}^{\tau -1}{\gamma }^{j-l}\varphi (s_{i,j},a_{i,j},s_{i,j+1})|s_{i,l}=s\right].\hfill \end{array}$$

(8)

The expectation in Equation (8) is taken over the actions taken from time step l and the states visited after time step l in an episode. We will consider the rest of the definitions from some ith agent’s perspective, $i\in \left[n\right]$, so subscripts i and $-i$ will be dropped for brevity.

We note that the action space and state space are finite and hence the set of strategies available to the players is also finite. The player adopts the lower myopic best response dynamics to choose the policy. A lower myopic policy selects an action with the lowest index among the actions that maximize the value function. As time proceeds, the strategies and the action distribution converge to the asymptotic equilibrium [10].

Let ${\alpha }_j^{\ast }\in {[0,1]}^{\left|\mathcal{A}\right|}$ be the limiting population action distribution for jth time index in episode k. We note that due to the episodic framework, the limiting action distribution depends on the index in an episode. Then, from the definition of limit, for every $ϵ>0$ there exist a $K_{ϵ}<\infty$ such that for all $k>K_{ϵ}$, we have

$$\left|\right|{\alpha }_{k\tau +j}-{\alpha }_j^{\ast }{\left|\right|}_2<ϵ$$

(9)

where ${\left|\right|.\left|\right|}_2$ denotes the ${\ell }_2$ norm.

The value function defined in Equation (8) satisfies the Bellman-property for finite horizon MDPs, given by

$$\begin{array}{cc}\hfill V_{\pi ,l}\left(s\right|{\alpha }_l^{\ast })& ={\mathbf{E}}_{P_i,{\pi }_i}\left[\sum _{j=l}^{\tau -1}{\gamma }^{j-l}{r}_{i,j}|s_{i,l}=s\right]\hfill \end{array}$$

(10)

$$\begin{array}{c}=\sum _{s^{\prime }\in S}{P}_i\left(s^{\prime }\right|s,a,{\alpha }_l^{\ast })r_{i,l}+{\mathbf{E}}_{P_i,{\pi }_i}\left[\sum _{j=l+1}^{\tau -1}{\gamma }^{j-l}{r}_{i,j}=s^{\prime }\right]\hfill \end{array}$$

(11)

$$\begin{array}{c}={\overline{r}}_l+\gamma \sum _{s^{\prime }\in S}P\left(s^{\prime }\right|s,a,{\alpha }_l^{\ast })V_{\pi ,l+1}\left(s^{\prime }\right|{\alpha }_{l+1}^{\ast })\hfill \end{array}$$

(12)

where ${\overline{r}}_l={\sum }_{s^{\prime }\in S}P\left(s^{\prime }\right|s,a,{\alpha }_l^{\ast })r_l$, and $a=\pi (s,l)$.

Similarly, we also define the Q-function as:

$$\begin{array}{cc}\hfill Q_{\pi ,l}(s,a|{\alpha }_l^{\ast })& ={\overline{r}}_l+\gamma \sum _{s^{\prime }\in S}P\left(s^{\prime }\right|s,a,{\alpha }_l^{\ast })V_{\pi ,l+1}\left(s^{\prime }\right|{\alpha }_{l+1}^{\ast })\hfill \end{array}$$

(13)

We further consider the agents are strategic and hence care only about individual rewards. The goal of each agent is to find an optimal oblivious policy ${\pi }^{\ast }$, such that,

$$\begin{array}{c}{V}_{{\pi }^{\ast },l}\left(s\right|{\alpha }_l^{\ast })\ge V_{\pi ,l}\left(s\right|{\alpha }_l^{\ast }) \forall s\in \mathcal{S}, \forall l\in \left[\tau \right].\hfill \end{array}$$

(14)

Let ${\alpha }^{\ast }=[{\alpha }_0^{\ast },\cdots ,{\alpha }_{\tau -1}^{\ast }]\in {[0,1]}^{\tau \times \left|\mathcal{A}\right|}$, then we can define the optimal oblivious strategy:

Definition 2.

The set $\mathcal{P}\left(\alpha \right)$ is the set of the optimal oblivious strategies which are chosen from the Q-function generated by α. In other words, for a given α, a policy $\overline{\pi }\in \mathcal{P}\left(\alpha \right)$ if and only if

$$\begin{array}{c}\hfill \overline{\pi }(s,l)\in \arg \underset{a}{\max }{Q}_{\overline{\pi },l}(s,a|{\alpha }_l)\forall s\in S l\in \left[\tau \right]\end{array}$$

(15)

Here, the policy $\overline{\pi }(s,l)$ is used at lth time index in an episode so that the Q-value $Q_{\pi ,l}\left(s\right)$ is maximized for all states $s\in \mathcal{S}$. Note that $\overline{\pi }(s,l)$ does not depend on the distribution ${\alpha }^{\ast }$ explicitly. Hence, it is an oblivious strategy where each agent takes its decision based on its own observed state only. Since the reward function is bounded and $\gamma <1$, the set $\mathcal{P}\left(\alpha \right)$ is always non-empty. However, finding the optimal action is challenging for an oblivious strategy profile. We denote the initial population state distribution denoted by $f_0$. We note that as ${\alpha }_t$ evolves, the population state distribution $f_t$ also evolves. After convergence, for a time index j in any episodes, the population state distribution will converge to the limiting population state distribution $f_j^{\ast }$, or

$$\begin{array}{c}\hfill \left|\right|f_{k\tau +j}-f_j^{\ast }{\left|\right|}_2\to 0.\end{array}$$

(16)

2.4. Stationary Mean-Field Equilibrium

Throughout this paper, we seek to compute a mean-field equilibrium and action strategy. Thus, the action was taken by an agent only depends on its state independent of its episode. Further, such an action profile should converge to a stationary action distribution and state distribution. We now formally define a mean-field equilibrium.

Definition 3

([24]). We say that Mean-Field Equilibrium (MFE) is achieved by an oblivious strategy $\overline{\pi }(·)$, if the strategy for the players, population action, and the state distribution is such that:

• Each player i optimizes its expected discounted payoff assuming that population action distribution α is fixed; i.e., it satisfies (15).
• For strategy $\overline{\pi }$ of any player i, the fixed population action distribution α satisfies

  $${\alpha }_j\left(a\right)=\frac{1}{n}\sum _s{\mathrm{𝟙}}_{\overline{\pi }(s^{\prime },j)=a}p\left(s^{\prime }\right|s,a,{\alpha }_{j-1}) \forall j\in \left[\tau \right]$$

  (17)
  We define the above as $\alpha \in \widehat{D}(\overline{\pi },f)$
• For α and $\overline{\pi }$, the state distribution f satisfies

  $$\begin{array}{c}\hfill f\left(s_j^{\prime }\right)=\sum _{s}f\left(s\right)p\left(s^{\prime }\right|s,\pi \left(s\right),{\alpha }_{j-1}) \forall j\in \left[\tau \right]\end{array}$$

  (18)

  which we denote as $f\in D(\overline{\pi },\alpha )$.

Specifically, if we fix $\alpha$ and each agent takes action $\overline{\pi }$ which belongs to $\mathcal{P}\left(\alpha \right)$, then the action distribution should return to $\alpha$ as it is invariant under the transition probability. Further, if we fix $\alpha$ and each agent takes action $\overline{\pi }$, the state distribution should give back f.

Since the players are learning an oblivious strategy, no agent observes the states or actions of the other agents. Also, an agent does not know the probability transition matrix, and reward function and will try to estimate it from the past observations as described in the next section.

3. Proposed Algorithm

In this section, we propose an algorithm, which will be shown to converge to the mean-field equilibrium (MFE) in the following section. For each agent i, the algorithm begins with a prior distribution g over the stochastic game with state-space set S and action space set A and time horizon $\tau$. The prior distribution g for modeling state transition probability distribution is typically taken to be Dirichlet distribution [14,15].

The game is played episodes $k=0,1,2,\cdots$. The length of each episode is given by $\tau$. In each episode, the game is played in discrete steps, $j=0,1,\cdots ,\tau -1$. The episodes begin at times $t_k=k\tau ,k=0,1,2,\cdots$. At each time $t=k\tau +j$, the state of the agent is given by $s_t$, it selects an action $a_t$, and observes a scalar reward $r_t$ then transitions to the state $s_{t+1}$. Let $H_t=(s_1,a_1,r_1,\dots s_t,a_{t-1},r_{t-1})$ denote the history of the agent till time t.

The proposed algorithm is described in Algorithm 1. At the beginning of each episode, the MDP, ${\mathcal{M}}_k$ is sampled from the posterior distribution conditioned on the history $H_{t_k}$ in Line 4. We note that the sampling of MDP only relates to the sampling of the transition probability P and the reward distribution since the rest of the parameters are known. We note that the algorithm doesn’t perform explicit exploration like an $ϵ$-greedy algorithm. Instead, the algorithm samples a new MDP ${\mathcal{M}}_k$ for episode k in Line 4. The Algorithm can generate a new trajectory from the new policy [14,15] solved for the sampled MDP ${\mathcal{M}}_k$. We assume that after some samples, ${\alpha }_k$ has converged. The proposed algorithm converges as the induced transition probability and reward function converge after ${\alpha }_k$ converge.

Algorithm 1 Proposed Algorithm for Mean-Field Game with Best Response Learning Dynamics.

1: Input: Prior distribution g, time horizon $\tau$, $\gamma$ 2: Initialize $H_0=\varphi$. 3: for episodes $k=0,1,2,\cdots$ do 4:       Sample ${\mathcal{M}}_k\sim g(·|H_{k\tau })$. 5:       Obtain optimal Q for ${\mathcal{M}}_k$ from Algorithm 2 6:       for time steps $j=0$, …, $\tau -1$ do 7:             Play $a_j=\arg {\max }_a{Q}_j(s_j,a)$. 8:             Observe reward $r_j$, action of the agent $a_j$, and next state $s_{j+1}$. 9:            Append action taken $a_j$, reward obtained $r_j$, and state update $s_{j+1}$ to history $$H_{k\tau +j+1}=H_{k\tau +j}\cup \{a_j,r_j,s_{j+1}\}.$$ 10:     end for 11: end for

We use Backwards Induction algorithm [25] described in Algorithm 2 to obtain the Q-value function for the current sampled MDP (Line 5, Algorithm 1). Backward induction in Algorithm 2 starts from the end of the episode and calculates the potential maximum rewards for each state and action (Line 5). The algorithm then goes back in the episode (Line 8), to calculate the maximum possible cumulative rewards for each state and action in Line 11. After all the time indices in an episode are covered, the algorithm returns the calculated optimal Q-values. We obtain the policy ${\pi }_k$ from the calculated Q-values and the policy is not altered in an episode. Recall for a given $\alpha$, a policy $\pi \in \mathcal{P}\left(\alpha \right)$ if and only if ${\pi }_k(s,j)\in \arg {\max }_a{Q}_{{\pi }_k,j}(s,a|{\alpha }_j)$ for all $s\in \mathcal{S}$ and $j=0,1,\dots ,\tau -1$. Let ${\alpha }_k$ be the population action distribution in episode k, then the algorithm aims to choose a policy ${\pi }_k\in \mathcal{P}\left({\alpha }_k\right)$. In order to choose the policy ${\pi }_k$ from the set $\mathcal{P}\left({\alpha }_k\right)$, we use lower myopic learning dynamics, where at each episode we choose the strategy which is the smallest action index in the set $\mathcal{P}\left({\alpha }_k\right)$.

Algorithm 2 Backwards Induction Algorithm.

1: Input: $\mathcal{M}=\{\mathcal{S},\mathcal{A},P,r,\tau ,\gamma \}$           ▹ Sampled MDP from Algorithm 1 2: Initialize $Q_l(s,a)=0 \forall s\in \mathcal{S},a\in \mathcal{A},l\in \left[\tau \right]$. 3: for state $s\in \mathcal{S}$ do 4:       for state $a\in \mathcal{A}$ do 5:            Update Q-value function for last action $$Q_{\tau -1}(s,a)=\sum _{s^{\prime }\in \mathcal{S}}P\left(s^{\prime }\right|s,a)r(s,a,s^{\prime })$$ 6:       end for 7: end for 8: for time steps $l=\tau -2,\cdots ,0$ do 9:       for state $s\in \mathcal{S}$ do 10:           for state $a\in \mathcal{A}$ do 11:               Update Q-value function $$Q_l(s,a)=\sum _{s^{\prime }\in \mathcal{S}}P\left(s^{\prime }\right|s,a)\times$$ $$\left(r(s,a,s^{\prime })+\gamma \arg \underset{a}{\max }{Q}_{l+1}(s^{\prime },a)\right)$$ (19) 12:           end for 13:     end for 14: end for 15: Return:  $Q_l(s,a) \forall l,s,a$

We note that ${\mathcal{M}}_k$ is used in the algorithm instead of ${\mathcal{M}}^{\ast }$ where ${\mathcal{M}}^{\ast }$, the true distribution, is not known. In order to obtain an estimate, each agent samples a transition probability matrix according to the posterior distribution. Each agent follows the strategy ${\pi }_k$ according to the Q-values over the episode. Based on the action decision by each agent, we update the value function and the Q-function based on the obtained reward functions which depend on the value of ${\alpha }_k$. The detailed algorithm steps can be seen in Algorithm 1. We note that, as the algorithm converges, the value of $\alpha$ converges, and thus all the transition probabilities and value functions depend on the limiting distribution.

4. Convergence Result

In this section, we’ll show that if the oblivious strategy is chosen according to the proposed algorithm, then the oblivious strategy $\pi$ and the limiting population action distribution $\alpha$ constitutes a Mean-Field Equilibrium (MFE). More formally, we have obtained the following–

Theorem 1.

The optimal oblivious strategy obtained from Algorithm 1 and the limiting action distribution constitute a mean-field equilibrium and the value function obtained from the algorithm converges to the optimal value function of the true distribution.

The rest of the section proves this result. We first note that the lower-myopic best response strategy leads to a convergence of the action strategy following the results in [10] for finite action space and state space. We note that there might be multiple actions that can maximize the state-action value function. This may lead to choosing different actions at different iterations for the same state. To avoid the oscillations between the best actions and hence keep the policy stable, we choose a lower-myopic strategy. This lower-myopic strategy avoids conflicts when the agents have a non-unique strategy that maximizes the value function. Further, any way of resolving the multiple optima could be used, including upper-myopic giving the same result. Having shown that $\alpha$ converges, we now proceed to show that the converged point of the algorithm results in an MFE.

We first show the conditions needed for a policy $\pi$, a population state f, and action distribution $\alpha$ to constitute an MFE (Section 4.1). Then, we show that the conditions for the policy to be MFE given in Section 4.1 are met for any optimal oblivious strategy (Section 4.2). Thus, the key property that is required to show the desired result is that the proposed algorithm leads to an optimal oblivious strategy. In order to show that, we show that the value function of the sampled distribution converges to the true distribution (Section 4.3). The result in Section 4.3 shows that the value function iterates eventually converge to the value function with knowledge of the true underlying distribution of the transition probability ${\mathcal{M}}^{\ast }$, thus proving that the proposed algorithm converges to an optimal oblivious strategy which constitutes a mean-field equilibrium thus proving the theorem.

4.1. Conditions for a Strategy to Be a MFE

In this section, we will describe the conditions for an oblivious strategy $\pi$ to be a MFE. Recall that, in Section 2.2, we defined two maps $\mathcal{P}\left(\alpha \right)$ and $\mathcal{D}(\pi ,\alpha )$. For a given action coupled stochastic game, the map $\mathcal{P}\left(\alpha \right)$ for a given population action distribution $\alpha$ gives the set of the optimal oblivious strategies. Further, the map $\mathcal{D}(\pi ,\alpha )$ for a given population action distribution $\alpha$ and oblivious strategy $\pi$ gives the set of invariant population state distribution f.

We define the map $\widehat{\mathcal{D}}(\pi ,f)$ which gives the induced population action distribution $\alpha$ induced from the oblivious strategy $\pi$ and the population state distribution f. The following lemma gives the conditions that the stochastic game constitutes a mean-field equilibrium. These conditions have been provided in [11], and the reader is referred to [11] for further details and proof of this result.

Lemma 1

(Definition 7 [11]). An action coupled stochastic game with the strategy π, population state distribution f and population action distribution α constitute a mean-field equilibrium if $\pi \in \mathcal{P}\left(\alpha \right),f\in \mathcal{D}(\pi ,\alpha )$ and $\alpha \in \widehat{\mathcal{D}}(\pi ,f)$.

4.2. Conditions of Lemma 1 Are Met for Any Optimal Oblivious Strategy

In this section, we show that the conditions of Lemma 1 are met for any optimal oblivious strategy. In the mean-field equilibrium, each agent plays according to the strategy $\pi \in \mathcal{P}\left(\alpha \right)$. If the average population action distribution is $\alpha$, and each agent takes an oblivious strategy, hence, we must have the evolution of the state space such that the oblivious strategy on those states leads to an average action distribution of $\alpha$. Since we assume a large number of agents, including agent i’s own state will not change the population state distribution. So, we let the average population state distribution be $f_j$ at time index j as,

$$\begin{array}{c}\hfill f_j\left(s\right)={\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}{\mathrm{𝟙}}_{s_{m,j}=s}\end{array}$$

(20)

where $s_{m,j}$ is the state of the agent m at time index j. Similarly, we also include agent i’s action as well in the population action distribution $\alpha$. Then, we have,

$$\begin{array}{cc}\hfill \mathbf{E}\left[f_j\left(s\right)\right]& =\mathbf{E}\left[{\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}{\mathrm{𝟙}}_{s_{m,j}=s}\right]\hfill \end{array}$$

(21)

$$\begin{array}{c}={\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}\mathbf{E}\left[{\mathrm{𝟙}}_{s_{m,j}=s}\right]\hfill \end{array}$$

(22)

$$\begin{array}{c}={\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}P\left(\{s_{m,j}=s\}\right)\hfill \end{array}$$

(23)

$$\begin{array}{c}={\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}\sum _{s^{\prime }\in \mathcal{S}}P\left(\{s_{m,j-1}=s\}\right)P\left(s|s^{\prime },\pi \left(s^{\prime }\right),{\alpha }_{j-1}\right)\hfill \end{array}$$

(24)

$$\begin{array}{c}=\sum _{s^{\prime }\in \mathcal{S}}\left({\displaystyle \frac{1}{n}}\sum _{m\in \left[n\right]}P\left(\{s_{m,j-1}=s\}\right)\right)P\left(s|s^{\prime },\pi \left(s^{\prime }\right),{\alpha }_{j-1}\right)\hfill \end{array}$$

(25)

$$\begin{array}{c}=\sum _{s^{\prime }\in \mathcal{S}}\mathbf{E}\left[f_{j-1}\left(s^{\prime }\right)\right]P\left(s|s^{\prime },\pi \left(s^{\prime }\right),{\alpha }_{j-1}\right)\hfill \end{array}$$

(26)

where $P\left(\{s_{m,j}=s\}\right)$ is probability of agent m being in state s at time index j. Here, Equation (24) follows from the transition probability matrix. Recursively replacing $f_{j-1}$ in Equation (26) using Equation (21) for all $j\in \left[\tau \right]$ gives the required result of $f\in \mathcal{D}(\pi ,\alpha )$.

The above statement also implies that $\alpha$ must satisfy

$$\begin{array}{c}\hfill {\alpha }_j\left(a\right)=\sum _{{\pi }^{-1}(a,j)}{f}_j\left(s\right)\end{array}$$

(27)

where ${\pi }^{-1}(a,j)$ represents the set of states s for which $a\in \pi (s,j)$. This is equivalent to saying that if all the agents follow the optimal oblivious strategy $\pi \in \mathcal{P}\left(\alpha \right)$, then the population state distribution f and the population action distribution $\alpha$ satisfy $f\in \mathcal{D}(\pi ,\alpha )$ and $\alpha \in \widehat{\mathcal{D}}(\pi ,f)$.

4.3. Sampling Does Not Lead to a Gap for Expected Value Function

In the last subsection, we proved that there exists an optimal oblivious strategy. We will show that the policies generated by Algorithm 2, ${\pi }_k$, for the sampled system dynamics ${\mathcal{M}}_k$ in episode k by Algorithm 1 converges to the optimal oblivious policy $\overline{\pi }$. To show the convergence of the policy ${\pi }_k$ to $\overline{\pi }$, we will show that the value function of the optimal oblivious policy ${\pi }_k$ converges to the optimal value function of the true system dynamics.

We will first describe the lemmas that are used to prove the required result. We start by stating the Azuma-Hoeffding Lemma for obtaining confidence intervals.

Lemma 2

(Azuma-Hoeffding Lemma [15]). If $Y_n$ is a zero-mean martingale with almost surely bounded increments, $|Y_i-Y_{i-1}|$ ≤ C, then for any δ≥ 0 with probability at least $1-\delta$, $Y_n\le C\sqrt{2n\log (1/\delta )}$.

We also utilize the following result of [15] on any $\sigma$-measurable function g.

Lemma 3.

If f is the distribution of ${\mathcal{M}}^{\ast }$ then, for any $\sigma \left(H_{t_k}\right)$-measurable function g,

$$\begin{array}{c}\hfill \mathbf{E}\left[g\left({\mathcal{M}}^{\ast }\right)\right|H_{t_k}]=\mathbf{E}\left[g\left({\mathcal{M}}_k\right)\right|H_{t_k}].\end{array}$$

(28)

At the start of every episode, each agent samples system dynamics from the posterior distribution given $H_{t_k}$. The following result bounds the difference between the optimal value function learned by the true distribution ${\mathcal{M}}^{\ast }$ function using the optimal policy ${\pi }^{\ast }$ which is unknown, and the optimal value function achieved by the sampled distribution ${\mathcal{M}}_k$ from the policy ${\pi }_k$.

Lemma 4.

Let $V_{{\pi }_k,j}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_j))$ be the optimal value function for an oblivious policy ${\pi }_k(s,j)=\arg \underset{a}{\max }{Q}_{{\pi }_k,j}^{{\mathcal{M}}_k}(s,a|{\alpha }_l)$ for the sampled system dynamics ${\mathcal{M}}_k$ chosen form Algorithm 1. Then $V_{{\pi }_k,j}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_j))$ converges to the optimal value function, $V_{{\pi }^{\ast },j}\left(s\right|{\alpha }_j)$, of the true system dynamics ${\mathcal{M}}^{\ast }$ i.e., for all states $s\in \mathcal{S}$ as $k\to \infty$ with probability at least $1-\delta$,

$$\begin{array}{c}\hfill V_{{\pi }_k,j}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_j))-V_{{\pi }^{\ast },j}\left(s\right|{\alpha }_j)\to 0\end{array}$$

(29)

Proof. 

We note that since the optimal value function is a $\sigma \left(H_{t_k}\right)$-measurable, we can use Lemma 3 to bound the difference of the optimal value functions of the sampled distribution at episode k and the true distribution to show that for all states $s\in \mathcal{S}$,

$$\begin{array}{c}\hfill \mathbf{E}\left[V_{{\pi }^{\ast },j}\left(s\right|{\alpha }_j)-V_{{\pi }_k,j}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_j))\right]=0\end{array}$$

(30)

Note that the length of all episodes is given by $\tau$ and the support of the reward is [0, 1]. Therefore for all states $s\in \mathcal{S}$, we have $V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0))\in [-\tau ,\tau ]$. Note that this condition is similar to bounded increments in Azuma-Hoeffding Lemma (Lemma 2).

Since $V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0)\in [-\tau ,\tau ]$ is a zero mean martingale with respect to the filtration $\{H_{t_k}:k=1,..,m\}$, and satisfies the assumptions of Azuma-Hoeffding Lemma, we obtain the result as in the statement of the Lemma. Also, for all states $s\in \mathcal{S}$, we have, $V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0))\in [-\tau ,\tau ]$. So, the difference is a zero-mean martingale and has the bounded increments property. Applying the Azuma-Hoeffding Lemma to the martingale, we have the following result,

$$\begin{array}{c}\hfill \sum _{k=1}^m\left(V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0)\right)\le \tau \sqrt{2m\log (1/\delta )}\end{array}$$

(31)

For total time T of the algorithm, we have $m=T/\tau$. Thus, we obtain,

$$\begin{array}{cc}\hfill \sum _{k=1}^m\left(V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0)\right)& \le \tau \sqrt{2\frac{T}{\tau }\log (1/\delta )}\hfill \end{array}$$

(32)

$$\begin{array}{c}=\sqrt{2T\tau \log (1/\delta )}\hfill \end{array}$$

(33)

Thus, for all $\theta >1/2$, as $T\to \infty$, we have

$$\begin{array}{c}\hfill \frac{{\sum }_{k=1}^{⌈T/\tau ⌉}\left(V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}{\left(s\right|\alpha )}_0\right)}{T^{\theta }}\to 0\end{array}$$

(34)

Substituting $\theta =1$, the above expression says that $\tau$ times the average difference in an episode which converges to zero as total time $T\to \infty$, which gives us the convergence of the optimal value functions of the two distributions. Thus, we have

$$\begin{array}{c}\hfill V_{{\pi }^{\ast },0}\left(s\right|{\alpha }_0)-V_{{\pi }_k,0}^{{\mathcal{M}}_k}\left(s\right|{\alpha }_0)\to 0 \mathrm{as} k\to \infty \end{array}$$

(35)

□

5. Conclusions

We consider an action-coupled stochastic game consisting of a large number of agents where the transition probabilities are unknown to the agents. We utilize the concept of mean-field equilibrium where each agent’s reward and the transition probability are only impacted through the mean distribution of the actions of the other agents. When the number of agents grows large, the mean-field equilibrium becomes equivalent to the Nash equilibrium. We propose a posterior sampling-based approach where each agent draws a sample using an updated posterior distribution and selects an optimal oblivious strategy accordingly. We show that the proposed algorithm converges to the mean-field equilibrium without knowing the transition probabilities apriori.

This paper shows asymptotic convergence to the mean-field equilibrium while finding the convergence rate is an interesting future direction. Further, the convergence rate in the number of users to the mean-field limit (akin to [26], while for the mean-field game rather than mean-field control) is an important direction.