Open Access This article is
- freely available
Risks 2018, 6(3), 90; https://doi.org/10.3390/risks6030090
Mean Field Game with Delay: A Toy Model
Department of Statistics & Applied Probability, University of California, Santa Barbara, CA 93106-3110, USA
Correspondence: [email protected]
Work supported by NSF grants DMS-1409434 and DMS-1814091.
Received: 12 July 2018 / Accepted: 20 August 2018 / Published: 1 September 2018
We study a toy model of linear-quadratic mean field game with delay. We “lift” the delayed dynamic into an infinite dimensional space, and recast the mean field game system which is made of a forward Kolmogorov equation and a backward Hamilton-Jacobi-Bellman equation. We identify the corresponding master equation. A solution to this master equation is computed, and we show that it provides an approximation to a Nash equilibrium of the finite player game.
Keywords:inter-bank borrowing and lending; stochastic game with delay; Nash equilibrium; master equation
MSC:91A15; 91G80; 60G99
A linear quadratic stochastic game model of inter-bank borrowing and lending was proposed in (Carmona et al. 2015). In this model, each individual bank tries to minimize its costs by controlling its rate of borrowing or lending to a central bank with no obligation to pay back its loan. The finding is that, in equilibrium, the central bank acts as a clearing house providing liquidity, and hence stability is enhanced. This model was extended in (Carmona et al. 2018), where a delay in the controls was introduced. The financial motivation is that banks are responsible for the past borrowing or lending, and need to make a repayment after a fixed time (the delay). In this model, the dynamics of the log-monetary reserves of the banks are described by stochastic delayed differential equations (SDDE). A closed-loop Nash equilibrium is identified by formulating the original SDDE in an infinite dimensional space formed by the state and the past of the control, and by solving the corresponding infinite dimensional Hamilton-Jacobi-Bellman (HJB) equation. For general stochastic equations and control theory in infinite dimension, we refer to (Bensoussan et al. 2007; Fabbri et al. 2017; Da Prato and Zabczyk 2008).
In this paper, we study the mean field game (MFG) corresponding to the model proposed in (Carmona et al. 2018) as the number of banks goes to infinity. We identify the mean field game system, which is a system of coupled partial differential equations (PDEs). The forward Kolmogorov equation describes the dynamics of the joint law of current state and past control, and the backward HJB equation describes the evolution of the value function. Recently, J.-M. Lasry and P.-L. Lions introduced the concept of “master equation” which contains all the information about the MFG. The well-posedness of this master equation in presence of a common noise and convergence of the N-player system is analyzed in (Cardaliaguet et al. 2015) by a PDE approach. A probabilistic approach is proposed in (Carmona and Delarue 2014; Chassagneux et al. 2014). See also the two-volume book (Carmona and Delarue 2018) for a complete account of this approach.
In this paper, the master equation for our delayed mean field game is derived, a solution is given explicitly, and we show that it is the limit of the closed-loop Nash equilibrium of the N-player game system as .
The paper is organized as follows. In Section 2, we briefly review the stochastic game model with delay presented in (Carmona et al. 2018). Then, in Section 3, we construct the corresponding mean field game system. In Section 4, we define derivatives with respect to probability measures in the space where is the Hilbert space defined at the beginning of Section 2.2. In addition, we derive the master equation, and exhibit an explicit solution. Furthermore, in Section 5, we show that this solution of the master equation is an approximation of order to the solution of the finite-player Nash system. Lastly, in Section 6, we compare the solution of the Nash system, the solution of the mean field game system, and the solution to the master equation.
2. A Differential Game with Delay
2.1. The Model
Let represents the log-monetary reserves of the N banks at time t. At each time t, bank i controls its rate of borrowing or lending , and it also needs to make a repayment after a fixed time such that , at a rate denoted by . The dynamic of log-monetary reserves for each bank is given bywith deterministic initial conditionswhere are independent standard Brownian motions, and banks have the same volatility .
Bank i interacts with other banks by choosing its own strategy in order to minimize its cost functional , which involves the average of log-monetary reserves of all the other banks. The notation is a tuple of the with and , which represents all other banks’ control except bank i. The cost functional for bank is given by:where the running and terminal cost functions f and g are:
2.2. Construction of a Nash Equilibrium
In order to apply the dynamic programming principle to identify a closed-loop Nash equilibrium, we have to enlarge the state space by including the path of past controls, which lie in , the Hilbert space of square integrable real functions defined on , and write an infinite dimensional representation for our system. This evolution equation approach was initiated in (Vinter and Kwong 1981) under a deterministic control setting, and later was generalized in (Gozzi and Marinelli 2006) to a stochastic control problem.
Given , , and will denote the two components of the product space . The inner product on will be denoted by , and it is defined by
Therefore, the new state is denoted by , which corresponds to in the notation of the original system (1).
Bank i tries to minimize its cost functional defined by
After all other players have chosen their optimal strategies which minimize their cost functionals, player i’s value function is defined by
By dynamic programming principle, the value function must satisfy the following infinite dimensional HJB equation (see Fabbri et al. 2017 Chapter 2 for details):with terminal condition , where the operator is defined asand its domain is .
The adjoint of A is and is defined bywith domain .
The operator is defined bywhere is the Dirac measure at .
The adjoint of B is given by
The operator is defined by
The infinite dimensional representation of the original system (1) is given by
By minimizing the Hamiltonian in (7), the infimum can be computed, so that the optimal control is attained at
Assuming that each player follows its own optimal strategy , which forms a Nash equilibrium, the corresponding value function follows the HJB equation
After applying the definitions of the operators and Q, the HJB equation for player i becomes:
As shown in (Carmona et al. 2018), a solution of the system (11) can be found in the formfor some deterministic functions , , , and satisfying the following PDEswith boundary conditions: and ,
3. The Mean Field Game System
The mean field game theory describes the structure of a game with infinite many indistinguishable players. All players are rational, i.e., each player tries to minimize their cost against the mass of other players. This assumption implies that the running cost and terminal cost in (4) only depend on i-th player’s state and the empirical distribution of . Denoting this empirical distribution bythese costs, as in (4), can be re-written as
As the number N of players goes to ∞, the joint empirical distribution of the states and past controlswith marginalsconverges to a deterministic limit denoted by (with marginals denoted by and ). Here, we assume that, at time 0, satisfies the LLN (for instance with i.i.d. ), and that the propagation of chaos property holds. A full justification of this property would involve generalizing the result in Section 2.1 of (Carmona and Delarue 2014) to an infinite dimensional setting in order to take into account the past of the controls. This is highly technical but intuitively sound. A complete proof is beyond the scope of this paper.
In the limit, a single representative player tries to minimize his cost functional, and, dropping the index i, his value function is defined assubject to
The HJB equation for the value function readswith terminal condition Then, we minimize in to get
After plugging it into (19), our backward HJB equation reads:
Next, since we “lift” the original non-Markovian optimization problem into a infinite dimensional Markovian control problem, we are able to characterize the corresponding generator for (18), which is denoted by ,where is a smooth function and the time dependency comes from given by (20). The derivation of the adjoint of is given in Appendix A. Consequently, the forward Kolmogorov equation for the distribution reads
Combining (21) with (23), we obtain the mean field game system. To solve this, We make the following ansatz for the value functionwhere we denote the mean of state , and the mean of past control . Plugging (24) into (23), multiplying both sides of (23) by , and integrating over , we have
After integration by parts, we obtainas can be seen directly using (24).
By integration by parts, we deduce
Now we are ready to verify the ansatz (24). We first compute the derivative of the ansatz,
Then, we plug the ansatz (24) into (7), and by collecting terms, terms, terms, and constant terms, we obtain the following system of PDEs:with boundary conditions
4. The Master Equation
The master equation for this delayed game lies in an infinite dimensional space, and it requires a notion of derivatives in the space of measures in .
The set of probability measure on is endowed with Monge-Kantorovich distancewhere is the collection of real-valued Lipschitz functions on with Lipschitz constant 1.
We say that is if there exists an operator such that for any and
If is of class with respect to , the marginal derivative is defined in the sense of Fréchet derivative:
Usually we will encounter a map . In this case, U can be expressed in a form of composition , where , and , i.e., .
If is with respect to , and is Fréchet differentiable, then , and are defined by
Suppose , where is Fréchet differentiable. Then can be written as , where , and . Then
Since , we have
4.2. The Master Equation
4.3. Explicit Solution of the Master Equation
It turns out that this master Equation (37) can be solved explicitly by making the following ansatz, and we also define and for convenience.
Then, we compute the partial derivatives needed in (37) explicitly, we haveand plug those into our master Equation (37). We have
5. Convergence of the Nash System
From the previous section, we have seen that our master equation is well posed, and we obtained an explicit solution. Furthermore, it also describes the limit of Nash equilibria of the N-player games as . In this section, generalizing to the case with delay the results of (Cardaliaguet et al. 2015) (see also Kolokoltsov et al. 2014), we show that the solution of the Nash system (11) converges to the solution of the master Equation (37) as number of players , with a Cesaro convergence rate.
In Section 4, we find that (40) is a solution to the master Equation (37). We set , where , denotes the joint empirical measure of and . The empirical measure of is given by , and the empirical measure of is given by . Note that, by direct computation, for , and any ,
For any , satisfieswhere , with terminal condition .
This shows that is “almost” a solution to the Nash system (11).
We first apply Ito’s formula to , and use the fact that satisfies the HJB Equation (11) for the Nash system.
Then, we apply Ito’s formula to , and use the fact that u satisfies (43)
Recall that is bounded by for , and is bounded by . Let be a family of independent random variable with common law . By integrating (48) from t to T, and taking expectation conditional on Ξ, we have
By the fact that , and using Young’s inequality, we have
Taking average on both sides, we have
By Gronwall’s inequality and taking supremum over , we havewhich implies
Choosing Ξ uniformly distributed in , then by continuity of and , and the fact that is defined by , we have, for any ,□
The mean field game system acts as a characteristic of the master equation. The master equation contains all the information in the mean field game system, and it turns the forward-backward PDE into a single equation. The solution to the mean field game system is a pair , that is the value function and the joint law of current state and past law. The solution to the master equation is a function of .
Since our model is linear quadratic, we are able to solve both the mean field game system and the master equation as shown in Section 3 and Section 4, however, the techniques are not the same. The technique for solving the mean field game is that we first make an ansatz for the solution of the HJB equation. Then plugging this ansatz into the Fokker-Planck Equation (23), we find that the means of state and past control are constant. Hence, the ansatz (24) can be verified. On the other hand, a notion of derivative with respect to measure is needed in order to solve the master equation. Again, we make an ansatz (40), which has a similar form as (24) but is a function of , and we verify that it satisfies the master equation.
The sets of PDEs (30) with boundary conditions (31) are the same for the two problems. This is due to the fact that our model is linear-quadratic and the means of states and past controls are constants.
Last but not the least, the Nash equilibrium of the corresponding N-player game is presented in Section 2. The value function (12) looks similar to the value function (24) in the mean field game system and the solution (40) to the master equation. As , the set of PDEs (13) becomes the same as (30). This implies that the solution to the mean filed game appears to be the limit of the Nash system, but generally, the convergence has been known in very few specific situations. Additionally, the solution to the master equation is also a limit to the Nash system, as shown in Section 5.
To summarize, we have extended the notion of master equation in the context of our toy model with delay, and we have shown that, as in the case without delay, this master equation provides an approximation to the corresponding finite-player game with delay. A general form of such a result, not necessarily for linear-quadratic games, is part of our ongoing research.
The two authors contributed equally to all aspects of the content of this article.
This research was funded by NSF grants DMS-1409434 and DMS-1814091.
Conflicts of Interest
The authors declare no conflict of interest.
The following abbreviations are used in this manuscript:
|SDDE||Stochastic Delayed Differential Equation|
|MFG||Mean Field Game|
|PDE||Partial Differential Equation|
|LLN||Law of Large Numbers|
|NSF||National Science Foundation|
Appendix A. Adjoint Operator
Let φ be a smooth test function defined on . In the following computation, we use the notation
If the test function φ is of the form for a smooth function ψ defined on , thenwhere is understood as a two-dimensional density. By abuse of notation, we also use
Then, we have
- Bensoussan, Alain, Giuseppe Da Prato, Michel C. Delfour, and Sanjoy Mitter. 2007. Representation and Control of Infinite Dimensional Systems. Basel: Birkhauser. [Google Scholar]
- Cardaliaguet, Pierre, Francois Delarue, Jean-Michel Lasry, and Pierre-Louis Lions. 2015. The master equation and the convergence problem in mean field games. arXiv, arXiv:1509.02505. [Google Scholar]
- Carmona, Rene, and Francois Delarue. 2014. The Master Equation for Large Population Equilibriums. arXiv, arXiv:1404.4694. [Google Scholar]
- Carmona, Rene, and Francois Delarue. 2018. Probabilistic Theory of Mean Field Games with Applications I & II. Berlin/Heidelberg: Springer International Publishing. [Google Scholar]
- Carmona, Rene, Jean-Pierre Fouque, Seyyed Mostafa Mousavi, and Li-Hsien Sun. 2018. Systemic risk and stochastic games with delay. Journal of Optimization and Applications (JOTA), 1–34. [Google Scholar] [CrossRef]
- Carmona, Rene, Jean-Pierre Fouque, and Li-Hsien Sun. 2015. Mean field games and systemic risk. Communications in Mathematical Sciences 13: 911–33. [Google Scholar] [CrossRef]
- Chassagneux, Jean-Francois, Dan Crisan, and Francois Delarue. 2014. A Probabilistic approach to classical solutions of the master equation for large population equilibria. arXiv, arXiv:1411.3009. [Google Scholar]
- Da Prato, Guiseppe, and Jerzy Zabczyk. 2008. Stochastic Equations in Infinite Dimensions. Cambridge: Cambridge University Press. [Google Scholar]
- Fabbri, Giorgio, Fausto Gozzi, and Andrzej Swiech. 2017. Stochastic Optimal Control in Infinite Dimension. Berlin/Heidelberg: Springer International Publishing. [Google Scholar]
- Gozzi, Fausto, and Carlo Marinelli. 2006. Stochastic optimal control of delay equations arising in advertising models. Stochastic PDEs and Applications VII 245: 133–48. [Google Scholar]
- Kolokoltsov, Vassili, Marianna Troeva, and Wei Yang. 2014. On the rate of convergence for the mean-field approximation of controlled diffusions with large number of players. Dynamic Games and Applications 4: 208–30. [Google Scholar] [CrossRef]
- Vinter, Richard B., and Raymond H. Kwong. 1981. The infinite time quadratic control problem for linear systems with state and control delays: An evolution equation approach. SIAM Journal on Control and Optimization 19: 139–53. [Google Scholar] [CrossRef]
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).