The Impact of Global Structural Information in Graph Neural Networks Applications

Abstract

Graph Neural Networks (GNNs) rely on the graph structure to define an aggregation strategy where each node updates its representation by combining information from its neighbours. A known limitation of GNNs is that, as the number of layers increases, information gets smoothed and squashed and node embeddings become indistinguishable, negatively affecting performance. Therefore, practical GNN models employ few layers and only leverage the graph structure in terms of limited, small neighbourhoods around each node. Inevitably, practical GNNs do not capture information depending on the global structure of the graph. While there have been several works studying the limitations and expressivity of GNNs, the question of whether practical applications on graph structured data require global structural knowledge or not remains unanswered. In this work, we empirically address this question by giving access to global information to several GNN models, and observing the impact it has on downstream performance. Our results show that global information can in fact provide significant benefits for common graph-related tasks. We further identify a novel regularization strategy that leads to an average accuracy improvement of more than $5\%$ on all considered tasks.

1. Introduction

Graph Neural Networks (GNNs) [1] are deep learning models for graph structured data, which achieve state-of-the-art results for many graph-related tasks. Most popular GNNs fall into the message-passing framework [2] and are denoted as Message Passing Neural Networks (MPNNs). (In this paper, we use the terms GNN and MPNN interchangeably.) MPNNs have become increasingly popular thanks to their simplicity, extensibility, and empirical effectiveness. MPNNs adopt a message passing mechanism where, at each layer, every node receives a message from its 1-hop neighbours. The incoming messages for each node are aggregated in a permutation-invariant fashion and used to update the node’s representation by the means of a learnable function (usually implemented with a neural network). The final node representations (also referred to as node embeddings) are then used to perform some graph-related downstream task, for example graph classification or node classification. Empirically, the best results are obtained when the message passing procedure is repeated a relatively small number of times (typical numbers are 2 to 5), as a higher number of layers leads to over-smoothing [3] and over-squashing [4]. Thus, practical GNNs are only leveraging the graph structure in the form of small neighbourhoods around each node. A direct consequence of this limitation is that GNNs are not capable of accessing, or extracting, information that depends on the whole structure of the graph (e.g., random walk probabilities [5]).

In this work, we are interested in studying the consequences of the over-smoothing and over-squashing issues. In more detail, we are interested in understanding whether global information (i.e., information that depends on the whole structure of the graph, and that cannot be recovered by just focusing on local neighbourhoods) is important for GNNs and their practical applications.

In fact, there is an ongoing debate in the GNN research community on whether it is needed to have “deep” GNNs [6], or if most tasks of interest only require access to local neighbourhoods. We tackle this question directly at its root, and address the overlooked aspect of whether global structural information is useful for GNN models, by studying if global structural information is important in practical scenarios. In more detail, we introduce three different ways to provide GNN models with global structural information, and study how they affect the performance of state-of-the-art MPNNs on common graph related tasks. The three strategies to include global structural information we consider are: (i) providing the model direct access to the adjacency matrix, (ii) providing the model direct access to random walks with restart coefficients, and (iii) combining (ii) with a regularization term which enforces the role of the information extracted by random walks with restart. These methods are introduced to study the impact of global information, and are not meant to be used as practical strategies to improve the performance of GNNs. On the latter aspect, we show that the sole use of our regularization term provides significant gains in performance while being easily and efficiently applicable to any GNN model. The use of random walks with restart is also supported by a theoretical contribution which proves they can increase the ability of GNNs in distinguishing non-isomorphic graphs.

• Our Contribution

Previous studies on the capabilities and limitations of GNNs have focused on the relation between GNNs and the Weisfeiler–Leman (WL) algorithm [7] to study the theoretical expressiveness of these models (e.g., [8]), or on how to alleviate the over-smoothing and over-squashing issues (e.g., [3,4,9]). There are, however, no empirical studies on the practical impact of global information (i.e., information that depends on the whole structure of the graph) in MPNNs.

We assess whether providing global information regarding the whole graph structure has a significant impact on the performance of state-of-the-art MPNNs. In this regard, our contributions are threefold.

• We propose and formalize three different types of global structural information “injection”. We test how the injection of global structural information impacts the performance of six GNN architectures (GCN [10], Graphsage [11], and GAT [12] for node-level tasks; GCN with global readout, DiffPool [13] and k-GNN [8] for graph-level tasks) on both transductive and inductive tasks. Results show that the injection of global structural information significantly impacts current state-of-the-art models on common graph-related tasks.
• As we discuss later in the paper, injecting global structural information can be impractical. We then identify a novel and practical regularization strategy, called RWRReg, based on random walks with restart [14]. RWRReg maintains the permutation-invariance of GNN models, and leads to an average $5\%$ increase in accuracy on both node classification and graph classification.
• We introduce a theoretical result proving that the information extracted by random walks with restart can “speed up” the 1-Weisfeiler–Leman (1-WL) algorithm [7]. In more detail, we show that, by constructing an initial coloring based on random walks with restart probabilities, the 1-WL algorithm always terminates in one iteration. Given the known relationship between GNNs and the 1-WL algorithm, this result shows that providing information obtained from random walks with restart to GNN models can improve their practical ability of distinguishing non-isomorphic graphs.

2. Preliminaries

In this section, we introduce the notation we use throughout the paper, and provide a brief introduction to GNNs and random walks with restart (RWR; also known as Personalized PageRank [14]).

2.1. Notation

We use uppercase bold letters for matrices ($\mathit{M}$), and lowercase bold letters for vectors ($\mathit{v}$). We use plain letters with subscript indices to refer to a specific element of a matrix ($M_{i,j}$), or of a vector ($v_i$). We refer to the vector containing the i-th row of a matrix with the subscript “$i,:$” (${\mathit{M}}_{i,:}$), while we refer to the i-th column with the subscript “$:,i$” (${\mathit{M}}_{:,i}$).

A graph $\mathcal{G}=(\mathcal{V},E)$, where $\mathcal{V}=\{1,..,n\}$ is the set of nodes and $E\subseteq \mathcal{V}\times \mathcal{V}$ is the set of edges, is represented by a tuple $(X,A)$. $X$ is an $n\times d$ matrix, where the i-th row contains the d-dimensional feature vector of the i-th node, and $A$ is the $n\times n$ adjacency matrix. For the sake of clarity, we restrict our presentation to undirected graphs, but similar concepts can be applied to directed graphs.

2.2. Graph Neural Networks

In graph representation learning, the goal is to learn a vector representation (also referred to as node embedding) for each node that can then be used to effectively perform downstream tasks. The message-passing framework [2], to which most GNNs belong, is based on the following procedure: each node receives messages from its neighbours, aggregates them, and updates its representation based on the aggregated messages and its previous representation. For a node v, with neighbours ${\mathcal{N}}_v$, we can represent the operations at the ℓ-th layer of message-passing as follows:

$$\begin{array}{cc}\hfill {\mathbf{m}}^{(v,\ell )}& =\mathrm{AGGREGATE}\left(\{{\mathbf{H}}_u^{(\ell )}\forall u\in {\mathcal{N}}_v\}\right)\hfill \\ \hfill {\mathbf{H}}_v^{(\ell +1)}& =\mathrm{UPDATE}({\mathbf{H}}_v^{(\ell )},{\mathbf{m}}^{(v,\ell )})\hfill \end{array}$$

where ${\mathbf{H}}^{(\ell )}$ is a matrix where the i-th row contains the representation of node i at layer ℓ, AGGREGATE is a permutation invariant function (e.g., average or sum) that takes as input the set of representations of the neighbours and aggregates them into a message $\mathbf{m}$, and UPDATE is usually a learnable function implemented with a neural network. The initial representation (at layer 0) is defined as ${\mathbf{H}}^{\left(0\right)}=\mathbf{X}$. As such, after k message-passing iterations, the representation of a node v depends on its k-hop neighbourhood (i.e., all the nodes at a distance of at most k from v). The GNNs proposed in literature differ on how they implement the AGGREGATE and UPDATE functions [1,10,11,12].

2.3. Random Walks with Restart

A RWR [14] for node i returns a vector ${\mathit{r}}^{\left(i\right)}$ of size n which satisfies the following equation:

$$r^{\left(i\right)}=(1-c)W{r}^{\left(i\right)}+c{e}^{\left(i\right)}$$

where $e^{\left(i\right)}$ is a vector where the i-th element is 1 and all the others are 0, c is the restart probability, and $W$ is the transition matrix of the random walk. The restart probability c defines the probability that the walk “jumps” back to the starting node (a common value for c, used in many libraries, is $0.15$). The RWR vector can be computed using the power iteration method, and over the year a large number of methods have been developed for its efficient and practical computation, or approximation, even for large scale graphs (e.g., [15,16]). Elements of $r^{\left(i\right)}$ capture the relative relationships between nodes [16], and the RWR vectors capture the global structure of the graph [17,18].

3. Random Walks with Restart and the Weisfeiler–Leman Algorithm

We provide analytical evidence that RWR can significantly empower MPNNs by proving a connection with the 1-Weisfeiler–Leman (1-WL) algorithm [7].

The 1-WL algorithm is a well known method for testing the isomorphism of two graphs. The 1-WL algorithm uses an iterative coloring, or relabeling, scheme, in which all nodes are initially assigned the same label (e.g., the value 1). It then iteratively refines the color of each node by aggregating the multiset of colors in its neighborhood with the use of a hash function. At every iteration, the feature representation of a graph is the histogram of resulting node colors. If, at a certain iteration of this process, two graphs have a different feature representation, then the two graphs are not isomorphic. (For a more detailed description of the 1-WL algorithm, we refer the reader to [7,19].)

It is known that not all non-isomorphic graphs are distinguishable by the 1-WL algorithm, and that n iterations are enough to distinguish two graphs of n vertices which are distinguishable by the 1-WL algorithm. There is a tight connection between 1-WL and MPNNs [10,20]. In particular, graphs that can be distinguished in k iterations by the 1-WL algorithm can be distinguished by certain GNNs in k message passing iterations [8]. This implies that, when using a GNN that can theoretically achieve the distinguishing power of the 1-WL algorithm, if such GNN is deployed with $k^{\prime }$ layers, it will not be able to distinguish graphs that are distinguishable by the 1-WL algorithm with $k^{\prime \prime }>k^{\prime }$ iterations.

Here, we prove that graphs that are distinguishable by 1-WL in k iterations have different feature representations extracted by RWR of length k, and hence if we use the RWR feature representations as initial coloring for the 1-WL algorithm, then the algorithm will always finish in one iteration. Given a graph $G=(V,E)$, we define its k-step RWR representation as the set of vectors ${\mathbf{r}}_v=[r_{v,u_1},\cdots ,r_{v,u_n}]$, $v\in V$, where each entry $r_{v,u}$ is the probability that a RWR of length k starting in v ends in $u\in V$.

Proposition 1.

Let $G_1=(V_1,E_1)$ and $G_2=(V_2,E_2)$ be two non-isomorphic graphs for which the 1-WL algorithm terminates with the correct answer after k iterations and starting from the labelling of all 1’s. Then, the k-step RWR representations of $G_1$ and $G_2$ are different.

The proof can be found in Appendix A. Since k iterations of the 1-WL algorithm are performed by MPNNs of depth k, but, in practice, MPNNs are limited to small depths; this result shows that RWR can empower MPNNs with relevant information that is discarded in practice.

We further provide an empirical analysis of RWR and their capability of encapsulating global information in Appendix F.

4. Injecting Global Information in MPNNs

To test if MPNNs are missing on important information that is encoded in the structure of a graph, we inject global structural information into existing MPNN models, and test how the performance of these models changes in several graph-related tasks. Intuitively, based on a model’s performance when injected with different types of global structural information, we can understand if this additional knowledge can improve performance on the considered tasks. In the rest of this section, we present the types of global structural information injection that we consider, and the models chosen for our experimental evaluation.

4.1. Types of Global Structural Information Injection

We consider three different types of global structural information injection described below. The injection strategies presented in this section are not designed for practical use, as the scope of these strategies is to help us understand the importance of global structural information. At this point, our objective is to study the impact of global structural information that is not accessible to GNN models. We discuss scalability and practical aspects in Section 6.

Adjacency Matrix. We provide GNNs with direct access to the adjacency matrix by concatenating each node’s adjacency matrix row to its feature vector. This explicitly empowers the GNN model with the connectivity of each node, and allows for higher level structural reasoning when considering a neighbourhood (the model will have access to the connectivity of the whole neighbourhood when aggregating messages from neighbouring nodes). In more detail, the row of the adjacency matrix for a specific node pinpoints the position of the node in the graph (i.e., it acts as a kind of positional encoding), and during the message passing procedure, when a node aggregates information from its neighbours, it allows the network to get a more precise positioning of the node in the graph.

Random Walk with Restart (RWR) Matrix. We perform RWR [14] from each node v, thus obtaining a n-dimensional vector that gives a score of how much v is “related” to every other node in the graph. For every node, we concatenate its vector of RWR coefficients to its feature vector. The choice of RWR is motivated by their capability to capture the relevance between two nodes [16] and the global structure of a graph [17,18], and by the possibility to modulate the exploration of long-range dependencies by changing the restart probability. Intuitively, if a RWR starting at node v is very likely to visit a node u (e.g., there are multiple paths that connect the two), then there will be a high score in the RWR vector for v at position u. This gives the GNN model higher level information about the global structure of the graph, and, again, it allows for high level reasoning on neighbourhood connectivity.

RWR Matrix + RWR Regularization. Together with the addition of the RWR score vector to the feature vector of each node, we also introduce a regularization term based on RWR that pushes nodes with mutually high RWR scores to have embeddings that are close to each other (independently of how far they are in the graph). Let $\mathit{S}$ be the $n\times n$ matrix with the RWR scores. We define the RWRReg (Random Walk with Restart Regularization) loss as follows:

$${\mathcal{L}}_{RWRReg}=\sum _{i,j\in V}{S}_{i,j}{\left||{\mathit{H}}_{i,:}-{\mathit{H}}_{j,:}|\right|}^2$$

where $H$ is a matrix of size $n\times d$ containing d-dimensional node embeddings that are in between message-passing layers (see Appendix B for the exact point in which $H$ is considered for each model). With this approach, the loss function used to train the model becomes: $\mathcal{L}={\mathcal{L}}_{original}+\lambda {\mathcal{L}}_{RWRReg}$, where ${\mathcal{L}}_{original}$ is the original loss function for each model, and $\lambda$ is a balancing term. In Appendix E, we show how to compute the RWRReg term efficiently using GPUs. We expect this type of information injection to have the highest impact on performance of the models on downstream tasks.

4.2. Choice of Models

In order to test the effect of the different types of global structural information injection and to obtain results that are indicative of the whole class of MPNNs models, we conceptually identify four different categories of MPNNs from which we select representative models.

• Simple Aggregation Models

Such models utilize a “simple” aggregation strategy, where each node receives messages (e.g., feature vectors) from its neighbours, aggregates them by assigning the same “importance” to each neighbour (e.g., by averaging their messages), and uses the aggregated messages to update its embedding vector. As a representative, we choose GCN [10], one of the fundamental and widely used GNNs models. We also consider GraphSage [11], as it represents a different computation strategy where a set of neighborhood aggregation functions are learned, and a sampling approach is used for defining fixed size neighbourhoods.

• Attention Models

Several models have used an attention mechanism in a GNN scenario [12,21,22,23]. These methods differ from the previous category as they use an attention mechanism to assign a different “weight”, or “importance”, to each neighbour. As a representative, we focus on GAT [12], the first to present an attention mechanism over nodes for the aggregation phase, and one of the best performing models on several datasets. Furthermore, it can be used in an inductive scenario.

• Pooling Techniques

Pooling on graphs is a very challenging task, since it has to take into account the underlying graph structure. At a high level, pooling methods provide a coarsened version of the input graph by combining groups of nodes into clusters. Among the methods that have been proposed for differentiable pooling on graphs [13,24,25,26,27], we choose DiffPool [13] for its strong empirical results. Furthermore, it can learn to dynamically adjust the number of clusters (the number is a hyperparameter, but the network can learn to use fewer clusters if necessary).

• Beyond WL

Morris et al. [8] prove that message-passing GNNs cannot be more powerful than the 1-WL algorithm, and propose k-GNNs, which rely on a subgraph message-passing mechanism and are proven to be as powerful as the k-WL algorithm. Another approach that goes beyond the WL algorithm was proposed by Murphy et al. [28]. Both models are computationally intractable in their initial theoretical formulation, so approximations are needed. As a representative, we choose k-GNNs, to test if subgraph message-passing is affected by additional global structural information.

5. Evaluation of the Injection of Global Structural Information

We now present our framework for evaluating the effects of the injection of global structural information into GNNs and the results of our experiments. Code for our method can be found at: https://github.com/DavideBuffelli/RWRReg, 9 January 2022. We consider one transductive task (node classification) and two inductive tasks (graph classification, and triangle counting). We use each architecture for the task that better suits its design: GCN, GraphSage, and GAT for node classification, and DiffPool and k-GNN for graph classification. We add an adapted version of GCN for graph classification, as a common strategy for this task is to deploy a node-level GNN, and then apply a readout function to combine node embeddings into a global graph embedding vector.

With regard to datasets, for node classification, we considered the three most used benchmarking datasets in literature: Cora, Citeseer, and Pubmed [29]. Analogously, for graph classification, we chose three frequently used datasets: ENZYMES, PROTEINS, and D&D [30]. Dataset statistics can be found in Appendix C.

For all the considered models, we take the hyperparameters from the implementations released by the authors. The only parameter tuned using the validation set is the balancing term $\lambda$ when RWRReg is applied. We found that the RWRReg loss tends to be larger than the Cross Entropy loss for prediction, and the best values for $\lambda$ lie in the range $[{10}^{-9},{10}^{-6}]$. For all the RWR-based techniques, we used a restart probability of $0.15$ (we use 0.15 as it is a common default value used in many papers and software libraries). The effects of different restart probabilities are explored in Section 6.) Detailed information on our implementations can be found in Appendix B.

5.1. Node Classification

For each dataset, we follow the approach that has been widely adopted in literature: we take 20 labeled nodes per class as training set, 500 nodes as validation set, and 1000 nodes for testing. Most authors have used the train/validation/test split defined by [31]. Since we want to test the general effect of the injection of global structural information, we differ from this approach and we do not rely on a single split. We perform 100 runs, where at each run we randomly sample 20 nodes per class for training, 500 random nodes for validation, and 1000 random nodes for testing. We then report mean and standard deviation for the accuracy on the test set over these 100 runs.

Results are summarized in

Table 1. Node classification accuracy results of different models with added Adjacency matrix features (AD), RWR features (RWR), and RWR features + RWR Regularization (RWR + RWRReg).

Model	Structural	Dataset
	Information	Cora	Pubmed	Citeseer
	none	$0.799\pm 0.029$	$0.776\pm 0.022$	$0.663\pm 0.095$
	AD	$0.806\pm 0.035$	$0.779\pm 0.070$	$0.653\pm 0.104$
GCN	RWR	$0.817\pm 0.025$	$0.782\pm 0.042$	$0.665\pm 0.098$
	RWR + RWRReg	$\mathbf{0.842}\pm \mathbf{0.026}$	$\mathbf{0.811}\pm \mathbf{0.037}$	$\mathbf{0.690}\pm \mathbf{0.102}$
	none	$0.806\pm 0.017$	$0.807\pm 0.016$	$0.681\pm 0.021$
	AD	$0.803\pm 0.014$	$0.803\pm 0.013$	$0.688\pm 0.020$
GraphSage	RWR	$0.816\pm 0.014$	$0.807\pm 0.015$	$0.693\pm 0.019$
	RWR + RWRReg	$\mathbf{0.837}\pm \mathbf{0.015}$	$\mathbf{0.820}\pm \mathbf{0.010}$	$\mathbf{0.728}\pm \mathbf{0.020}$
	none	$0.815\pm 0.021$	$0.804\pm 0.011$	$0.664\pm 0.008$
	AD	$0.823\pm 0.019$	$0.796\pm 0.014$	$0.672\pm 0.017$
GAT	RWR	$0.833\pm 0.020$	$0.811\pm 0.009$	$0.686\pm 0.009$
	RWR + RWRReg	$\mathbf{0.848}\pm \mathbf{0.019}$	$\mathbf{0.828}\pm \mathbf{0.010}$	$\mathbf{0.701}\pm \mathbf{0.011}$

, where we observe that the simple addition of RWR features to the feature vector of each node is sufficient to give a performance gain (up to 2%). The RWRReg term then significantly increments the gain (up to 7.5%). These results show that, perhaps surprisingly, even for the task of node classification global structural information is important.

5.2. Graph Classification

Following the approach from [8,13], we use 10-fold cross validation, and report mean and standard deviation of the accuracy on graph classification. Results are summarized in

Table 2. Graph classification accuracy results of different models with added Adjacency matrix features (AD), RWR features (RWR), and RWR features + RWR Regularization (RWR + RWRReg).

Model	Structural	Dataset
	Information	ENZYMES	D&D	PROTEINS
	none	$0.570\pm 0.052$	$0.755\pm 0.028$	$0.740\pm 0.035$
	AD	$0.591\pm 0.076$	$0.779\pm 0.022$	$0.775\pm 0.042$
GCN	RWR	$0.584\pm 0.055$	$0.775\pm 0.023$	$0.784\pm 0.034$
	RWR + RWRReg	$\mathbf{0.616}\pm \mathbf{0.065}$	$\mathbf{0.790}\pm \mathbf{0.023}$	$\mathbf{0.795}\pm \mathbf{0.032}$
	none	$0.661\pm 0.031$	$0.793\pm 0.022$	$0.813\pm 0.017$
	AD	$0.711\pm 0.027$	$0.837\pm 0.020$	$0.821\pm 0.039$
DiffPool	RWR	$0.687\pm 0.025$	$0.824\pm 0.028$	$0.783\pm 0.043$
	RWR + RWRReg	$\mathbf{0.721}\pm \mathbf{0.039}$	$\mathbf{0.840}\pm \mathbf{0.024}$	$\mathbf{0.834}\pm \mathbf{0.038}$
	none	$0.515\pm 0.111$	$0.756\pm 0.021$	$0.763\pm 0.043$
	AD	$0.572\pm 0.063$	$0.778\pm 0.020$	$0.751\pm 0.034$
k-GNN	RWR	$\mathbf{0.573}\pm \mathbf{0.077}$	$\mathbf{0.794}\pm \mathbf{0.022}$	$0.781\pm 0.028$
	RWR + RWRReg	$0.571\pm 0.080$	$0.786\pm 0.021$	$\mathbf{0.785}\pm \mathbf{0.026}$

. The performance gains given by the injection of global structural information are even more apparent than for the node classification task. Intuitively, this is explained by the fact that the global structure of the nodes in a graph is important for distinguishing different graphs. Most notably, the addition of the adjacency features is sufficient to give a large performance boost (up to 11%).

Surprisingly, models like DiffPool and k-GNN show an important difference in accuracy (up to 10%) when there is injection of structural information, meaning that even the most advanced methods suffer from the inability to exploit global structural information.

5.3. Counting Triangles

The TRIANGLES dataset [32] is composed of randomly generated graphs, where the task is to count the number of triangles contained in each graph. This is a hard task for GNNs and, as in [32], we use node degrees as node features to impose some structural information in the network. The TRIANGLES dataset has a test set with 10,000 graphs, of which half are similar in size to the ones in the training and validation sets (4–25 nodes), and half are bigger (up to 100 nodes). This permits an evaluation of a model’s capability generalization to graphs of unseen sizes.

For this regression task, we use a three layer GCN, and we minimize the Mean Squared Error (MSE) loss (more details can be found in Appendix B).

Table 3. Mean Squared Error of GCN with different types of global structural information injection on the TRIANGLES dataset.

Model	TRIANGLES Test Set
	Global	Small	Large
GCN	$2.290$	$1.311$	$3.608$
GCN-AD	$4.746$	$1.162$	$5.971$
GCN-RWR	$2.044$	$\mathbf{1.101}$	$2.988$
GCN-RWR + RWRReg	$\mathbf{2.029}$	$1.166$	$\mathbf{2.893}$

presents MSE results on the test dataset as a whole and on the two splits separately. We see that the addition of RWR features and of RWRReg provides significant benefits (up to 19% improvements), especially when the model has to generalize to graphs of unseen sizes, while the addition of adjacency features leads to overfitting (we provide more details in Appendix D).

6. Practical Aspects

From the results shown in Section 5, it would be tempting to propose the addition of adjacency matrix information or RWR information into node feature vectors as a strategy to improve the performance of GNN models. However, the benefits introduced by such a strategy come at a high cost: adding n features increases the input size of $n\times n$ elements (which is prohibitive for large graphs). Furthermore, all the considered models have a weight matrix at each layer that depends on the feature dimension, which means we are also increasing the number of parameters at the first layer by $n\times d^{\left(1\right)}$ (where $d^{\left(1\right)}$ is the dimension of the feature vector for each node after the first GNN layer). In this section, we propose a practical way to take advantage of the injection of global structural information without increasing the number of parameters, and controlling the memory consumption during training.

6.1. RWRReg

From Section 5, the use of RWR coefficients as additional features coupled with the additional RWRReg term is the strategy that provides the highest performance improvement on all tasks. As discussed at the beginning of this section, the addition of RWR coefficients can be problematic, and hence we study the impact of using only the RWRReg term. We consider the same settings and tasks presented in Section 5, and results are shown in

Table 4. Results for the addition of only the RWRReg term to existing models on node classification (accuracy), graph classification (accuracy), and triangle counting (MSE—lower is better).

Model	Regularization	Dataset
		Node Classification
		Cora	Pubmed	Citeseer
GCN	none	$0.799\pm 0.029$	$0.776\pm 0.022$	$0.663\pm 0.095$
	RWRReg	$\mathbf{0.861}\pm \mathbf{0.025}$	$\mathbf{0.799}\pm \mathbf{0.034}$	$\mathbf{0.686}\pm \mathbf{0.096}$
GraphSage	none	$0.806\pm 0.017$	$0.807\pm 0.016$	$0.681\pm 0.021$
	RWRReg	$\mathbf{0.841}\pm \mathbf{0.016}$	$\mathbf{0.818}\pm \mathbf{0.017}$	$\mathbf{0.721}\pm \mathbf{0.021}$
GAT	none	$0.815\pm 0.021$	$0.804\pm 0.011$	$0.664\pm 0.008$
	RWRReg	$\mathbf{0.824}\pm \mathbf{0.022}$	$\mathbf{0.811}\pm \mathbf{0.013}$	$\mathbf{0.702}\pm \mathbf{0.013}$
		Graph Classification
		ENZYMES	D&D	PROTEINS
GCN	none	$0.570\pm 0.052$	$0.755\pm 0.028$	$0.740\pm 0.035$
	RWRReg	$\mathbf{0.621}\pm \mathbf{0.041}$	$\mathbf{0.786}\pm \mathbf{0.024}$	$\mathbf{0.785}\pm \mathbf{0.036}$
DiffPool	none	$0.661\pm 0.031$	$0.793\pm 0.022$	$0.813\pm 0.017$
	RWRReg	$\mathbf{0.733}\pm \mathbf{0.032}$	$\mathbf{0.822}\pm \mathbf{0.025}$	$\mathbf{0.820}\pm \mathbf{0.038}$
k-GNN	none	$0.515\pm 0.111$	$0.756\pm 0.021$	$0.763\pm 0.043$
	RWRReg	$\mathbf{0.582}\pm \mathbf{0.075}$	$\mathbf{0.787}\pm \mathbf{0.022}$	$\mathbf{0.780}\pm \mathbf{0.028}$
		Triangles Test Set
		Global	Small	Large
GCN	none	$2.290$	$1.311$	$3.608$
	RWRReg	$\mathbf{2.187}$	$\mathbf{1.282}$	$\mathbf{3.014}$

. The results show that the sole addition of the RWRReg term increases the performance of the considered models by more than 5%. At the same time, RWRReg (i) does not increase the input size or the number of parameters, (ii) does not require additional operations at inference time, (iii) does not require additional supervision (it is in fact a self-supervised objective), (iv) maintains the permutation invariance of MPNN models, and (v) there is a vast literature on efficient methods for computing RWR, even for web-scale graphs (e.g., [15,33,34]). Hence, the only downside of RWRReg is the storage of the RWR matrix during training on very large graphs.

6.2. Sparsification of the RWR Matrix

To tackle the issue of storing in memory large RWR matrices, we explore how the sparsification of the RWR matrix affects the regularization of the model. In particular, we apply a top-K strategy: for each node, we only keep the K highest RWR weights. This approach can further take advantage of existing efficient methods to directly compute only the top-K RWR weights (e.g., [33,34,35,36]). As an example, TopPPR [33] provides guarantees on the precision of the returned scores, and requires only 15 seconds to retrieve the top-500 scores on a billion edge graph.

Figure 1 shows how different values of K impact performance on node classification (which usually is the task with the largest graphs). We can see that the addition of the RWRReg term is always beneficial. Furthermore, by taking the top-$\frac{n}{2}$, we can reduce the number of entries in the RWR matrix of $\frac{n^2}{2}$ elements, while still obtaining an average 3.2% increment on the accuracy of the model. This strategy then allows the selection of the value of K that best suits the available memory, while still obtaining a high performing model (better than GCN without global structural information injection).

6.3. Impact of RWR Restart Probability

The use of RWR requires to set the restart probability parameter. We show how performance changes with different restart probabilities. Intuitively, higher restart probabilities might put more much focus on close nodes, as the random walker with frequent return to the starting node. On the other side, lower probabilities allow for more long-range exploration, but may get “trapped” into densely connected subgraphs. Intuitively, we would expect lower probabilities to provide more information that is not already available to practical GNNs, and hence lead to higher performance. Figure 2 summarises how the accuracy on node classification (side (a)) and graph classification (side (b)) changes with different restart probabilities. (We did not go below $5\%$ for Cora, and $10\%$ for D&D for stability reasons in the computation of the RWR coefficients.) In accordance with our intuition, higher restart probabilities focus on close nodes (and less on distant nodes), and produce lower accuracies. Furthermore, we notice how injecting RWR information is never detrimental to the performance of the model without any injection.

7. Related Work

The field of GNNs has become extremely vast; for a thorough review, we refer the reader to a recent survey on the subject [1]. To the best of our knowledge, there are no studies that test if global information regarding the whole graph can significantly impact MPNNs on real-world tasks. However, there are some works that are conceptually related to our approach.

Several works have taken advantage of RWR in the context of MPNNs. Klicpera et al. [37] use RWR to create a new (weighted) adjacency matrix where message passing is performed. Li et al. [3] use random walks in a co-training scenario to add new nodes for the MPNNs’ training set. Ying et al. [38] and Zhang et al. [39] use random walks to define aggregation neighbourhoods that are not confined to a fixed distance. Abu-El-Haija et al. [40,41] use powers of the adjacency matrix, which can be considered as random walk statistics, to define neighbourhoods of different scales. Zhuang and Ma [42] use random walks to define the positive pointwise mutual information (PPMI) matrix and then use it in place of the adjacency matrix in the MPNN formulation. Klicpera et al. [43] use a diffusion strategy based on RWR instead of aggregating information from neighbours. This last work has recently been extended by [44] to scale to large graphs using RWRs to sample neighbourhoods. We remark how the aforementioned works focus on creating novel MPNN models, while we are interested in studying the impact of global structural information (which MPNNs do not have access to).

Gao et al. [45] and Ref. [46] uses regularization techniques to enforce that the embeddings of neighbouring nodes should be close to each other. The first uses Conditional Random Fields, while the second uses a regularization term based on the graph Laplacian. Both approaches only focus on 1-hop neighbours and do not take global information into account.

With regard to the study of the capabilities and weaknesses of GNNs, Refs. [3,47] study the over-smoothing problem that appears in Deep-GCN architectures, while Refs. [8,20] characterize the relation to the Weisfeiler–Leman algorithm. Other works have expressed the similarity with distributed computing [48,49], and the alignment with particular algorithmic structures [50]. These important contributions have advanced our understanding of the capabilities of GNNs, but they do not analyze or quantify the impact of global structural information.

Our RWRReg term relies on the computation of the RWR coefficients for every node (for computing the loss function). When dealing with large graphs, there is a vast literature on fast approximations of RWR scores [15,16,33,34,51,52].

Recent work [53] has shown that anonymous random walks (i.e., random walks where the global identities of nodes are not known) of fixed length starting at node u are sufficient to reconstruct the local neighborhood within a fixed distance of a node u [53]. Subsequently, anonymous random walks have been introduced in the context of learning graph representations [54]. Such results are complementary to ours, since they assume access to the distribution of entire walks of a given length, while our RWR representation only stores information on the probability of ending in a given node. In addition, such works do not provide a connection between RWR and 1-WL.

8. Conclusions

Whether global structural information (i.e., information that depends on the structure of the whole graph) is needed in GNNs for common tasks on graph-structured data is an open question. In this work, we tackle this question directly at its root. In particular, we identify three strategies to inject global structural information into MPNN models, and we quantify their impact on popular downstream tasks. Our experiments show that the additional information significantly boosts the performance of all considered state-of-the-art models, highlighting and quantifying the importance that global structural information can have on common MPNN applications. We further discuss a novel practical regularization technique based on RWR, which leads to an average improvement of $5\%$ on all models, and is supported by a novel connection between RWR and the 1-Weisfeiler–Leman algorithm.