Graph Attention Diffusion Method Combining Diffusion Mechanism and Graph Attention Mechanism

Abstract

Graph neural networks have attracted much attention and performed well in many downstream tasks. However, due to issues such as oversmoothing, existing graph neural networks are limited in their ability to quantitatively exploit higher-order neighborhood information. This paper introduces GAtD (Graph Attention Diffusion Method), which propagates attention to a wider range and aggregates higher-order information. We theoretically analyze the effectiveness of GAtD and demonstrate the convergence and linear complexity. A series of experiments demonstrates that, by combining diffusion and attention mechanisms, our method can effectively capture deep level relationships between nodes.

1. Introduction

Graph representation learning is a method of learning to use graph structured data, which can represent complex raw data as low dimensional vectors suitable for processing. It plays an important role in various fields, such as biomedicine [1,2] and road traffic [3,4]. In recent years, with the development of neural network technology, graph neural networks (GNNs) have attracted the interest of researchers, which has led to a series of remarkable advances [5,6,7].

GAT (Velikovi et al. [8]) is a representative method that cleverly combines attention mechanisms with graph convolution operations. There are already many outstanding new methods/applications based on GAT, such as SuperGAT [9], Social BiGAT [10], Kgat [11], etc. However, these methods operate within a limited neighborhood range, where the receptive field of a single attention layer is restricted to first-order neighbors. Although it is theoretically possible to obtain a larger receptive field by stacking multiple attention layers, it performs poorly in practice due to the issue of oversmoothing [12,13,14].

On the other hand, there have been many works dedicated to solving the oversmoothing problem, such as deepGCN [15], deeperGCN [16], PPNP [17], GDC [18]. However, these methods exclusively utilize graph convolution models, which means that only qualitative (structural) relationships are utilized, while quantitative (attention) relationships still need to be further explored.

Present work. In this paper, we propose a novel Graph Attention Diffusion (GAtD) framework that effectively diffuses attention to a wider range, thereby quantitatively aggregating information from higher-order neighborhoods without worrying about oversmoothing. At the same time, we theoretically analyze the relationship between GAtD and personalized page ranking (PPR) and use two different graph attention mechanisms that can be applied in different situations. We also analyze the computational complexity and convergence properties of GAtD, demonstrating its efficiency and theoretical guarantees. In addition, we conduct experiments on various real-world datasets, and the experimental results show that GAtD effectively captures and leverages deep level information through diffusing attention across the graph structure. The main contributions can be summarized as follows.

We propose GAtD, which can effectively diffuse attention to a wider range and quantitatively aggregate information from higher-order neighborhoods. Additionally, it is worth noting that GAtD is a universal framework that is not limited to the attention mechanisms used in the article but can be combined with any attention mechanism.

We theoretically analyze the relationship between GAtD and personalized page ranking (PPR), obtain a practical approximation algorithm, and analyze the convergence and computational complexity of GAtD.

We conduct experiments on various real-world datasets, and GAtD achieves superior performance compared to the baseline methods. In addition, we conduct ablation experiments to provide a more in-depth analysis and offer a comprehensive perspective.

2. Related Work

In this part, we focus on previous work closely related to our method. First, some notations are presented: let $G=(V,E,X,Y)$ denote a graph, where $V$ is the node set, and $E\subseteq (V\times V)$ is the edge set. $X$ denotes the node feature matrix, and $Y$ denotes the node label matrix. Let $A\in \{0,1{\}}^{n\times n}$ denote the adjacency matrix of $G$, where $n=\left|V\right|$ is the number of nodes.

Graph Neural Networks (GNNs). GNNs have extended deep learning methods to graph structured data [19,20] and have played an important role in many fields [21,22,23,24]. GNN generally follows a two-stage operation of “aggregation” ($\mathrm{A}\mathrm{g}\mathrm{g}(·)$) and “update” ($\mathrm{U}\mathrm{p}(·)$), that is, first obtaining “messages” from neighbor nodes and then updating the central node:

$$h_{N\left(u\right)}^{\left(k+1\right)}=\mathrm{A}\mathrm{g}{\mathrm{g}}^{\left(k\right)}\left(h_u^{\left(k\right)},h_v^{\left(k\right)}\right),v\in N\left(u\right),$$

(1)

$$h_u^{\left(k+1\right)}=\mathrm{U}{\mathrm{p}}^{\left(k\right)}\left(h_u^{\left(k\right)},h_{N\left(u\right)}^{\left(k+1\right)}\right),$$

(2)

where $k=0,1,\dots ,K$ represents the $k$-th layer of the GNN network, $h_u^{\left(k\right)}$ represents the embedding of node $u$ in $k$-th layer, and $N\left(u\right)$ is the neighbor node set of node $u$.

GAT [8] and APPNP [17]. GAT introduces the attention mechanism into GNNs, and its computational complexity is $\mathrm{O}(|V|\times d_1\times d_2+|E|\times d_2)$, where $d_1$ is the number of input features, and $d_2$ is the number of output features (Velikovi et al. [8]).

The aggregation and update of node messages are as follows:

$${\alpha }_{i,j}={\displaystyle \frac{\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(<\overrightarrow{a},{Wh}_i\left|\right|W{h}_j>\right)\right)}{{\sum }_{v\in N\left(i\right)}\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(<\overrightarrow{a},{Wh}_i\left|\right|{Wh}_v>\right)\right)}},$$

(3)

where ${\alpha }_{i,j}$ represents the attention coefficient between node $i,j$. $\overrightarrow{a}$ and $W$ represent the attention parameter vector and convolution parameter matrix, respectively, and $<x,y>$ represents the inner product of $x$ and $y$.

APPNP decouples the compression operation separately and then diffuses the message on the graph in a “jump back” manner (return to the source node), which can make the information of the node spread remotely without additional parameters. The general form of APPNP is as follows:

$$Z^{\left(0\right)}=H=f_{\theta }\left(X\right),$$

(4)

$$Z^{\left(k\right)}=\left(1-\alpha \right)P{Z}^{\left(k-1\right)}+\alpha H,k=1,2,\dots ,K,$$

(5)

where $f_{\theta }$ stands for the neural network, with $\theta$ as the donating parameter, $P$ is the information diffusion matrix, $\alpha$ is the restart parameter, and $K$ is the number of diffusion layers.

In addition, the Multi-hop Attention Graph Neural Network (MAGNA) [25] has a similar approach to our method, which combines a graph attention mechanism and a diffusion model to obtain long-range attention on the graph. However, these two methods have key differences. Firstly, the diffusion method of MAGNA is equivalent to computing the power of attention between nodes, while GAtD only calculates the attention between nodes once and uses the graph structure to propagate the attention to distant places. In other words, GAtD directly spreads attention on the graph, which reduces the computational complexity of GAtD and the complex operations between parameters, making it more conducive to optimization. Secondly, due to the more direct method of computing high-order attention in GAtD, it can flexibly adopt different attention mechanisms. In summary, GAtD adopts a lightweight architecture and can serve as a plug-and-play module. MAGNA and other diffusion-based graph Transformer models are more suitable for resource-intensive scenarios.

3. Methods

3.1. Graph Attention Diffusion (GAtD)

The global overview of GAtD is shown in Figure 1. Specifically, GAtD first projects the node representation $x_i$ into a latent space using Multilayer Perceptron (MLP) to obtain the latent representation $h_i$:

$$h_i=\mathrm{M}\mathrm{L}\mathrm{P}\left(x_i\right).$$

(6)

Then, we calculate the attention ${\alpha }_{i,j}$ between any neighbor nodes $i,j$ in the latent space (Section 3.1.1). Note that we calculate attention weights only between neighboring nodes, as shown in Figure 1A (taking the red node $i$ as the center node as an example).

Next, using the adjacency matrix to diffuse attention on the graph (Section 3.1.2), the deep level attention relationship ${\alpha }_{i,j}^{\prime }$ between nodes is obtained as shown in Figure 1B, and features are aggregated based on the final attention weights as shown in Figure 1C:

$$h_i^{\prime }=\sum _{j\in N\left(i\right)\cup i}{{\alpha }_{i,j}^{\prime }h}_j,$$

(7)

where $N\left(i\right)$ is the neighbor set of node $i$. Finally, the predicted vector is obtained:

$$z_i=softmax\left({\mathrm{M}\mathrm{L}\mathrm{P}(H}^{\prime }\right){)}_i,$$

(8)

where $h_i^{\prime }$ is the $i$-th line of $H^{\prime }$, the MLP and $softmax$ function project the node representations into the label space, and finally, we calculate the loss based on the label $y_i$ and conduct training:

$$loss=-\sum _v\left(y_v·\log \left(z_v\right)\right),v\in \mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}\mathrm{s}\mathrm{e}\mathrm{t},$$

(9)

where $y_v$ is the label vector (one-hot encoded), and $z_v$ is the predicted vector.

See Appendix A Algorithm A1 for the specific algorithm, and the details are introduced as follows.

3.1.1. Graph Attention

In this subsection, we introduce in detail the way to calculate attention between nodes.

After obtaining the latent representation $h_i$ of node $i$, we first calculate the attention ${\alpha }_{i,j}$ between nodes $i$ and $j$. Here, we implement two attention mechanisms Model I and $\mathrm{I}\mathrm{I}$ as follows:

$${\alpha }_{i,j}={\displaystyle \frac{\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({<h}_i,h_j>\right)\right)}{{\sum }_{v\in N\left(i\right)\cup i}\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({<h}_i,h_v>\right)\right)}},$$

(Model I)

$${\alpha }_{i,j}={\displaystyle \frac{\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(<\overrightarrow{a},h_i\left|\right|h_j>\right)\right)}{{\sum }_{v\in N\left(i\right)\cup i}\exp \left(\mathrm{L}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{y}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(<\overrightarrow{a},h_i\left|\right|h_v>\right)\right)}},$$

(Model II)

where $\overrightarrow{a}$ is the parameter vector, and $x\left|\right|y$ represents the concatenation of vectors $x$ and $y$.

As we can see, no additional parameters are used in the attention mechanism Model I, and only one parameter vector is introduced in the attention mechanism Model II; so, the parameters of GAtD are of the same magnitude as GCN.

In addition, GAtD uses a multi-head mechanism to obtain focus points in different latent spaces to increase stability, namely:

$${\alpha }_{i,j}^t=f\left(h_i^t,h_j^t\right),t=1,2,\dots ,T,$$

(10)

where $f$ represents some kind of attention mechanism, and $T$ represents the number of different latent spaces.

It is worth noting that our method GAtD is a general framework that can be combined with any attention mechanism. In this paper, we implement the above two attention mechanisms as examples.

3.1.2. Attention Diffusion

After calculating the attention between neighbor nodes, we get the attention matrix $A_{att}$, and then, we use the following set of formulas to diffuse the attention and get the deep attention relationship between nodes:

$$A^{\left(0\right)}=A_{att},$$

(11)

$$A^{\left(1\right)}=\left(1-\alpha \right)·A^{\left(0\right)}+\alpha ·I,$$

(12)

$$A^{\left(k\right)}=\left(1-\alpha \right)·\widehat{\tilde{A}}{A}^{\left(k-1\right)}+\alpha ·I,k=2,3,\dots ,K,$$

(13)

$$A^{\prime }=A^{\left(K\right)},$$

(14)

where $\alpha \in (0,1]$ is the jump-back probability, $I$ is the identity matrix, $\widehat{\tilde{A}}={\tilde{D}}^{-{\displaystyle \frac{1}{2}}}(A+I){\tilde{D}}^{-{\displaystyle \frac{1}{2}}}$ is the normalized adjacency matrix with self-loops, $K\ge 1$ is the number of diffusion operations performed, and $A^{\prime }$ is the final deep attention matrix. Finally, the aggregation operation is performed over all nodes covered by the diffusion attention matrix to integrate multi-hop neighbor information.

It can be seen that the existence of the jump-back probability $\alpha$ guarantees that the attention matrix $A^{\left(K\right)}$ does not become a zero matrix. When $k=1$, it is equivalent to using the attention matrix $A_{att}$ for feature aggregation; when $k>1$, GAtD directly uses the graph structure for attention diffusion, which reduces the complexity (see Complexity Analysis).

In addition, by combining the diffusion mechanism with the attention mechanism and incorporating self-loops, the attention diffusion mechanism enables GAtD to capture deep structural dependencies through attention diffusion, while controlling convergence via a jump-back probability $\alpha$ (see Convergence analysis).

3.2. Theoretical Analysis

Relationship with Personalized PageRank. The original PageRank can be obtained by calculating ${\pi }_{pr}=A_{pr}{\pi }_{pr}$, with $A_{pr}=A{D}^{-1}$, where ${\pi }_{pr}$ represents the limiting distribution [26]. Considering ${\pi }_{pr}$ as information and ignoring nonlinearity, we get the way GCN propagates information. Now, consider personalized PageRank with a jump-back probability $\alpha$, and its corresponding limiting distribution satisfies following formula:

$${\pi }_{ppr}=\left(1-\alpha \right)·A_{pr}{\pi }_{ppr}+\alpha ·e,$$

(15)

where $e$ represents uniform distribution. From this, we can get

$${\pi }_{ppr}=\alpha (I-\left(1-\alpha \right)·A_{pr}{)}^{-1}·e.$$

(16)

Observe Formula $\left(15\right)$: when $\widehat{\tilde{A}}$ is used instead of $A_{pr}$, and the distribution ${\pi }_{ppr}$ is considered as each column of the attention matrix (the initial distribution at this time is the one-hot vector of the corresponding node), it is consistent with the attention diffusion form of GAtD in Formula $\left(13\right)$. That is, the element ${\alpha }_{i,j}^{\prime }$ in the limit matrix $A^{\prime }$ (when $K\to \infty$ in Formula $\left(14\right)$) can be regarded as the influence of node $j$ on node $i$ at the level of attention features. Therefore, GAtD can be considered as a promising approach of the deep attention relationships between nodes.

Complexity Analysis. As described in Section 3.1, the GAtD method comprises five steps: feature projection, computation of initial attention, diffusion, aggregation and prediction. In the feature projection stage, the input is node feature matrix $X\in {\mathbb{R}}^{\left|V\right|\times d_1}$, the output is $H=X\times W_1{\in \mathbb{R}}^{\left|V\right|\times d_2}$, and the computational complexity is $\mathrm{O}(|V|\times d_1\times d_2)$.

In stage of computation of initial attention, the computational complexity is $\mathrm{O}(|E|\times d_2\times M)$ according to GAT [8], where $M$ is the number of attention heads. For the diffusion stage, we employ a technique in application: diffuse the projection matrix $H$ together with $A^{\left(k\right)}$. It is justified because the following iterative formula holds:

$$A^{\left(k\right)}H=\left(1-\alpha \right)\widehat{\tilde{A}}{A}^{\left(k-1\right)}H+\alpha H.$$

(17)

In each diffusion layer, only the matrix multiplication between $\widehat{\tilde{A}}$ and the matrix obtained in the previous layer needs to be computed, and the computational complexity of the diffusion stage is $\mathrm{O}(K|E|)$.

In the aggregation and prediction steps, the computational complexity is $\mathrm{O}(|V|\times d_2\times d_3)$, where $d_3$ is the dimension of the label vector.

Since $d_2,d_3,M$ are constants, and generally, we have $\left|E\right|\gg \left|V\right|$, the overall computational complexity simplifies as

$$\mathrm{O}\left(\left|V\right|\times d_1\times d_2\right)+\mathrm{O}\left(\left|E\right|\times d_2\times M\right)+\mathrm{O}\left(K\left|E\right|\right)+\mathrm{O}\left(\left|V\right|\times d_2\times d_3\right)=\mathrm{O}\left(\left|V\right|\times d_1+K\left|E\right|\right)=\mathrm{O}(K|E|).$$

(18)

In practice, $K$ is a hyperparameter; see the experiments section for detailed settings.

Convergence analysis. GAtD uses the iterative equation:

$$A^{\left(0\right)}=A_{att},$$

(19)

$$A^{\left(1\right)}=\left(1-\alpha \right)·A^{\left(0\right)}+\alpha ·I,$$

(20)

$$A^{\left(k\right)}=\left(1-\alpha \right)·\widehat{\tilde{A}}{A}^{\left(k-1\right)}+\alpha ·I,k=2,3,\dots ,K.$$

(21)

After $k$ diffusions we have

$$A^{\left(k\right)}={(1-\alpha )}^{\left(k-1\right)}{\widehat{\tilde{A}}}^{\left(k-1\right)}{A}^{\left(1\right)}+\alpha {\displaystyle \sum _{i=0}^{k-2}}(1-\alpha {)}^i{\widehat{\tilde{A}}}^i,k\ge 2.$$

(22)

We investigate the asymptotic behavior as $k\to \infty$.

For ${(1-\alpha )}^{\left(k-1\right)}{\widehat{\tilde{A}}}^{\left(k-1\right)}{A}^{\left(1\right)}$, $\widehat{\tilde{A}}$ is the normalized adjacency matrix with self-loops; so, all its characteristic values $\left|{\lambda }_i\right|\le 1$. Therefore, the spectral radius $\rho (\widehat{\tilde{A}})\le 1$. Since $\alpha \in (0,1]$, we have ${(1-\alpha )}^{\left(k-1\right)}{\widehat{\tilde{A}}}^{\left(k-1\right)}{A}^{\left(1\right)}\to 0$.

For ${\displaystyle {\sum }_{i=0}^{k-2}}(1-\alpha {)}^i{\widehat{\tilde{A}}}^i,k\ge 2$, the convergence condition is $\rho \left((1-\alpha )\widehat{\tilde{A}}\right)<1$. Given that $\rho (\widehat{\tilde{A}})\le 1$, it remains to prove that $\left(1-\alpha \right)<1$. Since $\alpha \in (0,1]$, ${\displaystyle {\sum }_{i=0}^{k-2}}(1-\alpha {)}^i{\widehat{\tilde{A}}}^i$ converges.

The characteristic values of $I-\left(1-\alpha \right)\widehat{\tilde{A}}$ are $I-\left(1-\alpha \right){\lambda }_i,(|{\lambda }_i|\le 1)$. Since $\alpha \in (0,1]$, $I-\left(1-\alpha \right){\lambda }_i\ne 0$. Therefore, matrix $I-\left(1-\alpha \right)\widehat{\tilde{A}}$ is invertible. Then, we get

$$A^{\left(\infty \right)}=\alpha (I-\left(1-\alpha \right)\widehat{\tilde{A}}{)}^{-1}.$$

(23)

In practical applications, typically, $K$ does not exceed 20. That is, the attention matrix $A^{\left(K\right)}$ is a weighted combination of the original attention matrix and the graph structure matrix and does not completely forget the original attention.

4. Experiments

4.1. Datasets

GAtD is evaluated on datasets from different fields (summarized in

Table 1. Summary of the datasets.

Datasets	Nodes	Edges	Features	Classes
Cora	2708	5429	1433	7
Citeseer	3327	4732	3703	6
Pubmed	19,717	44,338	500	3
CS	18,333	81,894	6805	15
Physics	34,493	247,962	8415	5
Chameleon	2277	36,101	2325	5

).

Cora, Citeseer and Pubmed [27]: Cora, Citeseer and Pubmed are three standard citation network datasets, where nodes represent documents, and edges (undirected) represent citation links. The three citation network datasets contain sparse feature vectors for each document, and each document has a category label. Dataset website: https://paperswithcode.com/datasets (accessed on 1 January 2025).

CS and Physics [28]: Physics and CS are co-authorship datasets based on the Microsoft Academic Graph of the 2016 KDD Cup Challenge. The nodes represent the authors, and connected nodes indicate that they are co-authors of the same paper. Node features are extracted from the keyword information of each author in the paper. Dataset website: https://github.com/shchur/gnn-benchmark#datasets (accessed on 1 January 2025).

Chameleon [29]: Chameleon is a network dataset of Wikipedia pages about chameleons, where nodes represent articles in Wikipedia, edges (directed) reflect links between web pages, and node features indicate the presence of specific nouns in the articles. Dataset website: https://github.com/radrumond/Chameleon (accessed on 1 January 2025).

These datasets are all general datasets for testing the performance of graph neural networks. We perform node classification tasks on these datasets to test the performance of GAtD. Next, we introduce the specific experimental settings and baseline methods.

4.2. Experimental Setup and Baselines

Our method has a simple structure, and the main hyperparameters are the same as GAT, except for $\alpha$ and $K$ (Section 3). For $\alpha$ and $K$, we perform a grid search in [0.05, 0.1, 0.2, 0.3, 0.4, 0.5] and [2, 5, 10, 15, 20], respectively. The other hyperparameters follow the settings in GAT [8].

Additionally, Cora, Citeseer and Pubmed employ the standard dataset split (following GCN [29]); for the CS and Physics datasets, 20 nodes per class are randomly sampled for the training set, 30 nodes per class for the validation set, and the remaining nodes for the test set; Chameleon follows the split in Geom-GCN [28].

In experiments on each dataset, 10 runs are performed, and the average accuracy and standard deviation are reported. During training, the maximum number of epochs is 200, with early stopping applied when the validation accuracy does not improve for 50 consecutive epochs. All baselines are evaluated under identical conditions.

Baselines. GAtD is compared with the following state-of-the-art methods, aside from MLP, GAT and APPNP:

• GCN [29] is a semi-supervised GNN model, which introduces the convolution operation.
• GraphSAGE [30] determines node neighborhoods by sampling and can generalize to unseen data.
• GIN [31] is a GNN framework with a simple structure which is also based on GCN. Its differentiation and representation capabilities can be comparable to the WL-test [32].
• GDC [18] also uses the idea of graph diffusion, which enhances GNNs using the generalized graph diffusion matrix.
• For each method, the hyperparameters follow their original optimal settings.

4.3. Results and Analysis

Main Results. We conduct 10 experiments on each dataset, and the experimental results are shown in

Table 2. Main results.

	Cora	Citeseer	Pubmed	CS	Physics	Chameleon
MLP	57.2 ± 1.03	56.9 ± 1.41	72.7 ± 0.71	87.5 ± 0.65	88.6 ± 0.54	51.6 ± 1.80
GCN	81.0 ± 0.33	70.9 ± 0.34	79.0 ± 0.26	89.4 ± 0.90	92.3 ± 0.76	62.4 ± 2.84
GraphSAGE	81.9 ± 0.57	70.9 ± 0.41	78.4 ± 0.32	86.2 ± 1.85	91.3 ± 0.82	62.9 ± 1.51
GIN	81.8 ± 0.61	70.4 ± 0.57	78.6 ± 0.44	89.6 ± 0.96	91.4 ± 1.53	58.8 ± 1.32
GAT	82.4 ± 0.59	71.4 ± 0.49	78.6 ± 0.28	89.9 ± 0.68	92.1 ± 0.88	58.9 ± 2.26
APPNP	82.6 ± 1.14	71.7 ± 0.91	79.2 ± 0.52	90.5 ± 0.33	90.3 ± 0.92	56.8 ± 3.47
GDC	82.3 ± 0.84	71.4 ± 0.75	79.6 ± 0.49	90.8 ± 1.13	91.7 ± 0.86	60.6 ± 2.37
GAtD-I (ours)	83.7 ± 0.73	72.3 ± 0.85	79.9 ± 0.23	92.9 ± 0.47	93.2 ± 0.57	64.7 ± 2.17
GAtD-II (ours)	84.0 ± 0.57	72.3 ± 0.82	79.8 ± 0.20	93.0 ± 0.37	93.1 ± 0.94	64.5 ± 2.55

The best result is denoted in “bold”, the second in “underline”, and the third in “wavy underline”.

. GAtD-I and GAtD-II represent the GAtD methods combined with attention mechanism Model I and attention mechanism Model II, respectively.

The results show the average accuracy ± standard deviation of node classification across 10 independent runs. The higher the accuracy and the smaller the standard deviation, the better the classification effect. Among them, the best result is denoted in “bold”, the second in “underline”, and the third in “wavy underline”. It can be seen that GAtD-I/GAtD-II achieves strong performance on all datasets, especially on the CS dataset, where GAtD has a 2.2% advantage over the best method in baselines. These results demonstrate the effectiveness of GAtD.

When the best effect is achieved, the corresponding hyperparameters ($\alpha$, $K$) are set as

Table 3. Hyperparameter ($\alpha$, $K$) settings of GAtD-I.

	Cora	Citeseer	Pubmed	CS	Physics	Chameleon
$\alpha$	0.1	0.05	0.1	0.5	0.2	0.05
$K$	15	20	15	10	15	2

and

Table 4. Hyperparameter ($\alpha$, $K$) settings of GAtD-II.

	Cora	Citeseer	Pubmed	CS	Physics	Chameleon
$\alpha$	0.1	0.1	0.1	0.5	0.3	0.05
$K$	20	15	20	10	20	2

.

It can be seen that, except for the Chameleon dataset, when the best effect is achieved, the number of diffusion layers of the other datasets is more than 10. At the same time, combined with the comparison with APPNP, the GAtD method can make full use of the diffusion mechanism and attention mechanism to capture the deep relationship between nodes.

In addition, compared with other datasets, it is noted that the jump-back probability $\alpha$ is significantly higher when the co-authored datasets CS and Physics achieve the best results, which suggests that the CS and Physics datasets pay more attention to the feature information of the node itself, while other datasets can benefit more from neighbor information.

Ablation Experiment. GAtD mainly focuses on two hyperparameters: jump-back probability $\alpha$ and diffusion number $K$. To this end, we further explore the impact of $\alpha$ and $K$ on the results.

First, we study how node classification accuracy changes with $\alpha$, and the results are shown in Figure 2.

Figure 2 shows how the performance of GAtD-I and GAtD-II changes with $\alpha$ on various datasets when $K$ remains unchanged ($K$ is fixed to the corresponding optimal setting). The horizontal axis represents the value of $\alpha$, and the vertical axis represents the node classification accuracy.

It can be seen that, on all datasets, GAtD-I and GAtD-II show regular trends, and their trends are consistent on all datasets, which show that the GAtD method can accurately model and learn the deep attention relationship between nodes in the graph (otherwise, the trend should be messy and not consistent on all datasets). In addition, it is noted that, on the Cora, CS, and Chameleon datasets, the accuracy fluctuates strongly (more than 5%), which may suggest that the node characteristics in these three graphs are quite different.

Then, we study how the accuracy changes with $K$, and the results are shown in Figure 3.

Figure 3 shows the performance of GAtD-I and GAtD-II as $K$ changes when $\alpha$ remains unchanged on each dataset. The horizontal axis represents the value of $K$, and the vertical axis represents the node classification accuracy.

As $K$ increases, GAtD achieves better performance on most datasets, demonstrating that it effectively alleviates the oversmoothing problem. Once again, we can see the regular trend of change, and the trends are consistent on all datasets, which supports the effectiveness of GAtD.

Finally, we show the calculation time of GAtD in Figure 4.

Figure 4 shows the calculation time of GAtD at different diffusion layers on Cora, Citeseer, and Pubmed. GAT is also shown as a baseline control, and the vertical axis represents time (in seconds).

It can be seen that as the number of diffusion layers increases, the calculation time of GAtD also increases (linearly). When the number of diffusion layers is 2 ($K=2$), the calculation time of GAtD is basically the same as that of GAT. These experimental phenomena verify the theoretical analysis in Section 3 (Complexity Analysis).

To ensure the reliability of performance improvements, we use a t-test to verify the statistical significance between GAtD and the suboptimal methods.

Table 5. T-test experimental results.

Datasets	GAtD vs. Suboptimal Methods	Accuracy Improvement	t-Statistic	p-Value
Cora	vs. APPNP	+1.4%	3.26	0.005
Citeseer	vs. APPNP	+0.6%	1.08	0.154
Pubmed	vs. GDC	+0.3%	2.00	0.038

shows the statistical results. GAtD significantly outperforms suboptimal methods on two out of three datasets. On Citeseer, although the difference does not reach statistical significance, GAtD achieves a higher average performance.

Additionally, we computed the 95% confidence intervals (

Table 6. 95% confidence intervals.

Datasets	GAtD	Suboptimal Methods
Cora	83.98 [83.55, 84.41] %	82.56 [81.70, 83.42] %
Citeseer	72.30 [71.68, 72.92] %	71.72 [71.03, 72.41] %
Pubmed	79.91 [79.74, 80.08] %	79.59 [79.22, 79.96] %

) to illustrate the precision of performance estimates. GAtD exhibits narrower confidence intervals, indicating higher stability.

5. Conclusions

In this paper, we propose a novel graph attention diffusion framework GAtD, which propagates attention to a wider range and aggregate higher-order information. Moreover, GAtD can be combined with any attention mechanism, and this paper provides two attention mechanisms as examples. In addition, we theoretically analyze the relationship between GAtD and personalized page ranking (PPR), demonstrate its effectiveness, and simultaneously analyze the convergence and computational complexity of GAtD. The final series of experiments supported our analysis, and the competitive performance demonstrates that GAtD can fully utilize diffusion and attention mechanisms to capture deep level relationships between nodes, making it a promising framework for deep level attention modeling.

Future work includes combining more types of attention mechanisms to adapt to data with different features or extending GAtD to more complex types of graphs.