Physics-Informed Graph Neural Operator for Mean Field Games on Graph: A Scalable Learning Approach

Abstract

Mean-field games (MFGs) are developed to model the decision-making processes of a large number of interacting agents in multi-agent systems. This paper studies mean-field games on graphs ($\mathcal{G}$-MFGs). The equilibria of $\mathcal{G}$-MFGs, namely, mean-field equilibria (MFE), are challenging to solve for their high-dimensional action space because each agent has to make decisions when they are at junction nodes or on edges. Furthermore, when the initial population state varies on graphs, we have to recompute MFE, which could be computationally challenging and memory-demanding. To improve the scalability and avoid repeatedly solving $\mathcal{G}$-MFGs every time their initial state changes, this paper proposes physics-informed graph neural operators (PIGNO). The PIGNO utilizes a graph neural operator to generate population dynamics, given initial population distributions. To better train the neural operator, it leverages physics knowledge to propagate population state transitions on graphs. A learning algorithm is developed, and its performance is evaluated on autonomous driving games on road networks. Our results demonstrate that the PIGNO is scalable and generalizable when tested under unseen initial conditions.

1. Introduction

Multi-agent systems (MAS) are prevalent in engineering and robotics applications. With a large number of interacting agents in the MAS, solving agents’ optimal control could be computationally intractable and not scalable. To solve this challenge, MFGs are [1,2] developed to model strategic interactions among many agents who make dynamically optimal decisions, while a population distribution is propagated to represent the state of interacting agents. Since its inception, MFGs have been widely applied to social networks [3], swarm robotics [4] and intelligent transportation [5,6].

MFGs are micro-macro games that bridge agent dynamics and population behaviors with two coupled processes: individuals’ dynamics solved by optimal control (i.e., agent dynamic) and system evolution arising from individual choices (i.e., population behaviors).

In this work, we focus on a class of MFGs [7], namely, mean-field games on graphs ($\mathcal{G}$-MFG), where the state space of the agent population is a graph and agents select a sequence of nodal and edge transitions with a minimum individual cost. Solving these $\mathcal{G}$-MFGs, however, poses the following challenges: (1) With a graph-based state space, the action space expands significantly, encompassing both nodes and edges, resulting in a high-dimensional search space. More specifically, the decision-making of a representative agent in $\mathcal{G}$-MFG consists of not only en-route choices at nodes but also continuous velocity control on edges subject to congestion effects. (2) Existing work mainly assumes that the initial population distribution is fixed. The change in initial population states leads to the re-computation of mean-field equilibria (MFE), a task that requires computational and memory resources and hinders the practicality of deploying MFG solutions.

To address these challenges, this paper proposes a new learning tool for $\mathcal{G}$-MFGs, namely, a physics-informed graph neural operator (PIGNO). The key element is a graph neural operator (GNO), which can generate population dynamics given the initial population distribution. To enhance the training process, the GNO incorporates physics knowledge regarding how agent and population dynamics propagate over the spatiotemporal domain.

Related Work

Researchers have explored various machine learning methods, such as reinforcement learning (RL) [8,9,10,11], and physics-informed neural networks (PINN) [12,13,14]. However, it can be time-consuming and memory-demanding for these learning tools to adapt to changes in initial population density. Specifically, each unique initial condition may require the assignment and retraining of a dedicated neural network to obtain the corresponding MFE. To enhance the scalability of the learning framework for MFGs, Chen et al. [15] introduced a physics-informed neural operator (PINO) framework. This framework utilizes a Fourier neural operator (FNO) to establish a functional mapping between mean-field equilibrium and boundary conditions. However, the FNO fails to solve $\mathcal{G}$-MFGs because it cannot directly project information over a graph into a high-dimensional space and generate population dynamics in the graph state space. Therefore, in this paper, we propose a graph neural operator (GNO) that learns mappings between graph-based function spaces to solve $\mathcal{G}$-MFGs. The GNO leverages message-passing neural networks (MPNNs) to handle state space and propagate state information efficiently by aggregating the neighborhood messages.

Our contributions include: (1) We propose a scalable learning framework leveraging PIGNO to solve $\mathcal{G}$-MFGs with various initial population states; (2) We develop a learning algorithm and apply it to autonomous driving games on road networks to evaluate the algorithm performance.

The rest of this paper is organized as follows: Section 2 introduces preliminaries about $\mathcal{G}$-MFGs. Section 3 presents the details of our scalable learning framework for $\mathcal{G}$-MFGs. Section 4 presents the solution approach. Section 5 demonstrates numerical experiments. Section 6 concludes.

2. Background

2.1. Mean-Field Games on Graphs ($\mathcal{G}$-MFG)

Mean-field games on graphs ($\mathcal{G}$-MFG) model population dynamics and a generic agent’s optimal control on both nodes and edges. A $\mathcal{G}$-MFG consists of a forward FPK and multiple backward HJB equations, which are defined on a graph $\mathcal{G}=\{\mathcal{N},\mathcal{L}\}$ as follows:

Definition 1. 

A $\mathcal{G}$-MFG with discrete time graph states [16] is:

$$\begin{array}{ccc}\left[\mathcal{G}\text{-}MFG\right]:\hfill & \hfill & \hfill \\ \hfill \left(FPK\right)& {\rho }^{\tau +1}={\left[P^{\tau }\right]}^T{\rho }^{\tau },{\rho }^0\equiv \tilde{\rho }\hfill & \left(1\mathrm{a}\right)\hfill \\ \hfill \left(HJB\right)& {\mathit{V}}^{\tau }=\underset{\mathit{u},\beta }{min}{P}^{\tau }{\mathit{V}}^{\tau +1}+{\mathit{r}}^{\tau },{\mathit{V}}^T\equiv \tilde{\mathit{V}}\hfill & \left(1\mathrm{b}\right)\hfill \end{array}$$

${\rho }^{\tau }={\left[{\rho }_{ij}^{\tau }\right]}^T$ is the population density on each edge $(i,j)\in \mathcal{L}$ at time step $\tau$. $\tilde{\rho }$ denotes the initial population density over the graph. The Fokker–Planck (FPK) equation captures the evolution of the population state on the graph. The Hamilton–Jacobi–Bellman (HJB) equation captures the optimal control of a generic agent, including the velocity control on edges and route choice on nodes. ${\mathit{V}}^{\tau }={\left[V_{ij}^{\tau }\right]}^T$ is the value function at each edge. $\tilde{\mathit{V}}$ denotes the terminal cost. ${\mathit{u}}^{\tau }={\left[u_{ij}^{\tau }\right]}^T$ denotes the exit rate at each edge, which represents the agent’s velocity control. ${\beta }^{\tau }={\left[{\beta }_{ij}^{\tau }\right]}^T$ is the probability of choosing node j as the next-go-to node at node i, i.e., route choice. ${\mathit{r}}^{\tau }={\left[r_{ij}^{\tau }\right]}^T$ is the cost incurred by the agent at time step $\tau$. The transition matrix $P^{\tau }$ is determined by ${\mathit{u}}^{\tau }$ and ${\beta }^{\tau }$. The MFE is denoted by $SOL\left(\left[\mathcal{G}\text{-}\mathrm{MFG}\right]\right)=\{{\rho }^{*},{\mathit{V}}^{*},{\mathit{u}}^{*},{\beta }^{*}\}$, satisfying Equation (1). The mathematical details of $\mathcal{G}$-MFG can be found in Appendix A.1. We provide a toy example in Appendix A.2 to help readers better understand it.

2.2. Graph Neural Operator (GNO)

Graph neural operators (GNOs) are generalized neural networks that can learn functional mappings between high-dimensional spaces [17]. GNO utilizes an MPNN to update space representation according to messages from the neighborhood. In this paper, we adopt a GNO to establish mappings between initial population state ${\rho }^0$ and population ${\rho }^{*}$ at MFE. We leverage the physics knowledge (i.e., FPK and HJB equations) to train the GNO for solving MFE with various initial population densities, eliminating the need to recompute MFE.

3. Scalable Learning Framework

In this section, we propose a physics-informed graph neural operator (PIGNO) to learn $\mathcal{G}$-MFGs. Figure 1 illustrates the workflow of two couple modules: FPK for population behaviors and HJB for agent dynamics. The FPK and the HJB modules internally depend on each other. In the FPK module, we estimate population density ${\rho }^{0:T}$ over the graph and update the GNO using a residual defined by the physical rule that captures population dynamics triggered by the transition matrix defined in the FPK equation. In the HJB module, the transition matrix $P^{0:T}$ is obtained given the population density ${\rho }^{0:T}$. We adopt another GNO to solve the HJB equation since the dynamics of the agents and the cost functions are known in the MFG system. We now delve into the details of the proposed PIGNO.

3.1. PIGNO for Population Behaviors

The GNO-$\rho$ maps the initial population distribution and the population distribution from time 0 to T. The input of GNO-$\rho$ is the initial population density ${\rho }^0$ along with the transition information to propagate population dynamics. The output of GNO-$\rho$ is the population dynamics over the spatiotemporal domain, denoted by $\widehat{\rho }\equiv {\widehat{\rho }}^{0:T}$. The PIGNO is instantiated as the following MPNN: $\forall (i,j)\in \mathcal{L},\tau =0,1,\dots ,T$

$${\widehat{\rho }}_{ij}^{\tau }={\rho }_{\theta }({\rho }_{ij}^0,\sum _{\forall (k,l)\in {\mathcal{L}}_{ij}^{\tau }}{\kappa }_{\varphi }({\rho }_{ij}^0,{\rho }_{kl}^0,e_{ij,kl}^{\tau }))$$

(2)

where, ${\widehat{\rho }}_{ij}^{\tau }$ is the population density of edge $(i,j)$ at time $\tau$, ${\mathcal{L}}_{ij}^{\tau }$ is the set of neighborhood edges of edge $(i,j)$ at time $\tau$, ${\kappa }_{\varphi }$ is the graph kernel function for ${\rho }_{\theta }$, and $e_{ij,kl}^{\tau }$ denotes the cumulative message used to propagate population dynamics from time 0 to time $\tau$. $e_{ij,kl}^{\tau }$ indicates the ratio of the population entering from edge $(k,l)$ to edge $(i,j)$ till time $\tau$, which is determined by the ratio of population exiting the edge $(k,l)$ (i.e., the velocity control u) and the ratio of the population choosing the edge $(i,j)$ as the next-go-to edge (i.e., the route choice $\beta$). The MPNN utilizes the initial population distribution and the message to propagate the population dynamics in the $\mathcal{G}$-MFG system. The neighborhood message is transformed by a kernel function ${\kappa }_{\varphi }$ and aggregated as an additional feature to estimate population density.

The GNO-$\rho$ adopts a physics-informed training scheme, which combines both model-driven and data-driven methods. The training of GNO-$\rho$ is guided by the residual determined by physical rules of population dynamics. Mathematically, the residual $r_{\theta }$ is:

$$\begin{array}{c}\hfill r_{\theta }=\frac{1}{|{\rho }^{\mathcal{D}}|}\sum _{{\rho }^0\in {\rho }^{\mathcal{D}}}{L}_{{\rho }^0},\end{array}$$

(3)

where, the set ${\rho }^{\mathcal{D}}$ contains various initial densities over the graph. $L_{{\rho }^0}$ is calculated as:

$$L_{{\rho }^0}={\alpha }_0·\frac{1}{T}\sum _{\tau =0}^{T-1}\left|\right|{\widehat{\rho }}^{\tau +1}-{\left[P^{\tau }\right]}^T{\widehat{\rho }}^{\tau }\left|\right|+{\alpha }_1·\left|\right|{\widehat{\rho }}^0-{\rho }^0{\left|\right|}^2,$$

(4)

where, the first term in $L_{{\rho }^0}$ evaluates the physical discrepancy based on Equation (1a). It integrates the residual of the FPK equation, ensuring that the model adheres to established laws of motion. When predicted $\rho$ becomes closer to ${\rho }^{*}$ satisfying the FPK equation, the residual gets closer to 0. The second term quantifies the discrepancy between the estimations and the ground truth of the initial density. The observed data comes from the initial distribution of population ${\rho }^0$. The training of ${\rho }_{\theta }$ based on observed data follows the traditional supervised learning scheme. ${\alpha }_0$ and ${\alpha }_1$ are the weight coefficients.

3.2. PIGNO for Optimal Control

Similar to GNO-$\rho$, GNO-V learns a reverse mapping from the terminal costs to the value functions over the graph from time T to 0. The input of GNO-V is the terminal costs ${\mathit{V}}^T$ and the transition information. The output of GNO-V is the value function over the spatiotemporal domain, denoted by $\widehat{\mathit{V}}\equiv {\widehat{\mathit{V}}}^{0:T}$. The GNO-V also follows the MPNN formulation: $\forall (i,j)\in \mathcal{L},\tau =0,1,\dots ,T$

$${\widehat{V}}_{ij}^{\tau }=V_{\eta }(V_{ij}^T,\sum _{\forall (k,l)\in {\mathcal{L}}_{ij}^{\tau }}{\kappa }_{\psi }(V_{ij}^T,V_{kl}^T,e_{ij,kl}^{\tau }))$$

(5)

where ${\widehat{V}}_{ij}^{\tau }$ is the value function of edge $(i,j)$ at time $\tau$, ${\mathcal{L}}_{ij}^{\tau }$ is the set of neighborhood edges of edge $(i,j)$ at time $\tau$, ${\kappa }_{\psi }$ is the graph kernel function for $V_{\eta }$, and $e_{ij,kl}^{\tau }$ denotes the cumulative message used to propagate population dynamics from time 0 to time $\tau$, which comes from the transpose of the cumulative transition matrix in GNO-$\rho$. The MPNN utilizes the terminal costs and the message to propagate system value functions. The neighborhood message is transformed by a kernel function ${\kappa }_{\psi }$ and aggregated as an additional feature to estimate value functions.

The training of GNO-V takes the HJB residual $r_{\eta }$ into consideration, which is

$$\begin{array}{c}\hfill r_{\eta }=\frac{1}{|{\mathit{V}}^{\mathcal{D}}|}\sum _{{\mathit{V}}^T\in {\mathit{V}}^{\mathcal{D}}}{L}_{{\mathit{V}}^T},\end{array}$$

(6)

where, the set ${\mathit{V}}^{\mathcal{D}}$ contains various terminal costs over the graph. $L_{{\mathit{V}}^T}$ is calculated as:

$$L_{{\mathit{V}}^T}={\alpha }_0·\frac{1}{T}\sum _{\tau =0}^{T-1}\left|\right|{\widehat{\mathit{V}}}^{\tau +1}-P^{\tau }{\widehat{\mathit{V}}}^{\tau }\left|\right|+{\alpha }_1·\left|\right|{\widehat{\mathit{V}}}^T-{\mathit{V}}^T{\left|\right|}^2,$$

(7)

where, the first term in $L_{{\mathit{V}}^T}$ evaluates the physical discrepancy based on Equation (1b). When predicted $\mathit{V}$ becomes closer to the optimal ${\mathit{V}}^{*}$, the residual gets closer to 0. The second term calibrates the predictions to the ground truth of the terminal costs ${\mathit{V}}^T$ by supervised learning. Similarly, ${\alpha }_0$ and ${\alpha }_1$ are the weight coefficients.

Note that a meta assumption of this model is discrete time. There are significant limitations of adopting a continuous-time model: (1) A continuum formulation of forward process over a graph space cannot be easily captured by several coupled partial differential equation systems. One way is to simplify the decision making over the graph as discretized route choice at each node, rendering the game as a continuous time markov decision process, which fails to capture the real-time velocity control on each edge. The other way is to formulate the dynamic process as a graph ODE, which requires further investigation on scalability. (2) A continuum formulation of backward process can be solved by continuous-time reinforcement learning. We leave the scalability of this continuous time reinforcement learning scheme to future work.

4. Solution Approach

In this section, we present our learning algorithm (Algorithm 1). We first initialize the GNO-$\rho$ ${\rho }_{\theta }$ parameterized by $\theta$ and GNO-V $V_{\eta }$ parameterized by $\eta$. During the ith iteration of the training process, we first sample a batch of initial population densities ${\rho }^0$ and terminal costs ${\mathit{V}}^T$. The terminal cost ${\mathit{V}}^T$ denotes the delay for agents who haven’t arrived at their destinations at the terminal time step. In this work, we assume the time delay at the terminal step is proportional to the travel distance between the agent’s location and her destination. ${\rho }^{\mathcal{D}}$ represents the set of initial population density. In this work, we interpret initial population density as the travel demands (i.e., the number of agents entering a graph at time 0). Agents can enter the graph from each node. We assume at time 0, the number of agents at each node (i.e., travel demand) follow a uniform distribution. We use each pair of ${\rho }^0$ and ${\mathit{V}}^T$ to generate the population density ${\widehat{\rho }}^{0:T}$ and ${\widehat{\mathit{V}}}^{0:T}$ over the entire domain. Given ${\widehat{\rho }}^{0:T}$ and ${\widehat{\mathit{V}}}^{0:T}$, we obtain the spatiotemporal transition P for all nodes. We then update the parameter $\theta$ of the neural operator according to the residual. At the end of each iteration, we check the convergence according to:

$$\begin{array}{c}\frac{{\sum }_{{\rho }^0\in {\rho }^D}|{\widehat{\rho }}^{0:T\left(i\right)}-{\widehat{\rho }}^{0:T(i-1)}|}{|{\rho }^D|}+\frac{{\sum }_{{\mathit{V}}^0\in {\mathit{V}}^D}|{\widehat{\mathit{V}}}^{0:T\left(i\right)}-{\widehat{\mathit{V}}}^{0:T(i-1)}|}{|{\mathit{V}}^D|}<ϵ\end{array}$$

(8)

Algorithm 1 PIGNO-MFG

1: Initialize: GNO-$\rho$ ${\rho }_{\theta }$ parameterized by $\theta$, GNO-V $V_{\eta }$ parameterized by $\eta$; 2: for $i\leftarrow 0$ to I do 3:     Sample a batch of initial population densities ${\rho }^0$ from the set ${\rho }^{\mathcal{D}}$ and terminal costs ${\mathit{V}}^T$ from the set ${\mathit{V}}^{\mathcal{D}}$; 4:     Predict ${\widehat{\rho }}^{0:T\left(i\right)}$ and ${\widehat{\mathit{V}}}^{0:T\left(i\right)}$ by ${\rho }_{\theta }$ and $V_{\eta }$ corresponding to each pair of ${\rho }^0$ and ${\mathit{V}}^T$ in the batch; 5:     for each ${\widehat{\rho }}^{0:T\left(i\right)}$ and ${\widehat{\mathit{V}}}^{0:T\left(i\right)}$ do 6:         Obtain $P^{0:T-1\left(i\right)}$ by solving the HJB. 7:     end for 8:     Obtain residual $r_{{\theta }^{\left(i\right)}}$ and $r_{{\eta }^{\left(i\right)}}$ according to Equations (3) and (6); 9:     Update ${\rho }_{\theta }$ and $V_{\eta }$; 10:     Check convergence (Equation (8)). 11: end for 12: Output MFE

5. Numerical Experiments

In this section, we employ our algorithm to facilitate autonomous driving navigation in traffic networks. As illustrated in Figure 2, a substantial number of autonomous vehicles (AVs) move to destination node 4, with the objective of minimizing total costs subject to the congestion effect. We use a representative agent as an example to elaborate on the speed control and density dynamics of the population in this scenario. At node 1, the representative agent first selects edge $l_{12}$. The agent then drives along edge $l_{12}$ and selects continuous-time-space driving velocities on the edge. The agent selects her next-to-go edge at node 2, following this pattern until she reaches her destination at node 4. These choices regarding her route and speed will actively influence the evolution of population density across the network. The mathematical formulation of this autonomous driving game can be found in [16].

We construct the initial population state over the network as follows: We assume that at time 0, the traffic network is empty. Vehicles enter the road network at origin nodes $1,2,3$ and move toward the destination 4. Travel demands at each origin satisfy $d_i\sim \mathrm{Uniform}[0,1],i=1,2,3$. Therefore, each initial population distribution over the network consists of travel demands at origins (i.e., $[d_1,d_2,d_3]$), which are sampled from three independent uniform distributions.

Figure 3 demonstrates the convergence performance of the algorithm in solving $\mathcal{G}$-MFG. The x-axis represents the iteration index during training, the y-axis displays the convergence gap, and the 1-Wasserstein distance measures the closeness between our results and the MFE obtained by numerical methods [16]. The results demonstrate that our algorithm can converge stably after 50 iterations.

Figure 4 demonstrates the population density solved by our proposed method along three paths on the road network, i.e., $(1\to 2\to 3\to 4)$, $(1\to 2\to 4)$, and $(1\to 3\to 4)$. The x-axis is the spatial position on the path, and the y-axis represents the time. The z-axis represents the population density $\rho$. The running cost functional form follows a non-separable cost structure with a crossing term of the agent action and the population density. We visualize the population density in $\mathcal{G}$-MFG with three initial population states, which are constructed by travel demands $[d_1,d_2,d_3]$: $[0.6,0.4,0.2]$ (See Figure 4a–c), $[0.4,0.4,0.4]$ (See Figure 4d–f) and $[0.2,0.4,0.6]$ (See Figure 4g–i). The plots show the population density evolution along each path on the graph. At the node, the density flow can be split when making route choice. For example, at node 2, vehicles can choose node 3 or 4 as their next node. It means the vehicle flow on the edge (1,2) is split into flows on edges (2,3) and (2,4).

6. Conclusions

In this paper, we propose a scalable learning framework, $\mathcal{G}$-MFGs. Existing numerical methods have to recompute MFE when the initial population density changes. To avoid recomputing MFE inefficiently, this work proposes a learning method, which utilizes graph neural networks to establish a functional mapping between the initial population density and MFE over the entire spatial temporal domain. This learning framework allows us to obtain MFEs corresponding to each initial population distribution without recomputing them. We demonstrate the efficiency of this method in autonomous driving games. Our contribution lies in the scalability of PIGNO to handle various initial population densities without recomputing MFEs. Our framework offers a memory- and data-efficient approach for solving $\mathcal{G}$-MFGs.