HGNN-AS: Enhancing Hypergraph Neural Network for Node Classification Accuracy with Attention and Self-Attention

Abstract

The incorporation of attention mechanisms into hypergraphs is an effective method for obtaining appropriate node and hyperedge representations. However, the ability of attention mechanisms to aggregate features on a hypergraph can be improved, particularly for noisy hypergraphs with connections between unrelated nodes that were constructed in a KNN-like manner. In this paper, we propose HGNN-AS, an enhanced hypergraph neural network model that achieves good accuracy on node classification tasks by combining an attention mechanism and a self-attention mechanism. Specifically, we introduce two self-attention mechanisms to improve how the HGNN-AS model expresses attention when distinguishing mislinked neighbours. Moreover, we add multihead attention mechanisms to our model to stabilize the training effect. The proposed model is evaluated on benchmark node classification tasks, including citation network classification and visual object recognition. Our experimental results demonstrate that our model outperforms most advanced methods on both tasks; among them, the accuracy improvement on the Cora dataset is the most noticeable, with an accuracy of 83.9%.

1. Introduction

Hypergraphs are generalized graph structures in which the edges are connected to an arbitrary number of nodes to represent higher-order relationships. Compared with general graph structures, hypergraphs can model multivariate relationships more accurately and extract richer information. With these characteristics, hypergraphs have demonstrated broad applicability across various domains, including protein complex modelling in bioinformatics [1], collaborative cyber-attack detection in network security [2], and complex relationship modelling in financial analysis [3].

Recently, hypergraphs have received considerable interest in the field of graph learning. Many works have proposed a series of neural networks for hypergraphs, which are collectively referred to as hypergraph neural networks. Among these methods, HGNN [4] was the first proposed hypergraph neural network model; this model realized information dissemination on hypergraphs through the spectral method. HyperGCN [5] is a semisupervised graph convolution training model based on spectral theory. HyperGAT [6] uses a dual attention mechanism to aggregate node features and hyperedge features in hypergraphs. The approaches used in these models for aggregating node and hyperedge features depend on the hypergraph structure.

However, hypergraph information cannot be used effectively. Hypergraphs are often noisy because hyperedges may include nodes that are semantically or categorically unrelated, which can mislead hypergraph neural networks and result in suboptimal node representations. HyperGAT adopts an attention mechanism to alleviate this issue. Similar to attention in sequential data [7,8,9] and graph data [10], hypergraph attention captures the relational importance of a hypergraph, that is, the degree of significance between each node on the hyperedge and the same node on different hyperedges. Although HyperGAT has an improved node classification performance, this method cannot prevent noise or extract richer information in the hypergraphs. This is a critical issue that should be addressed in the field of hypergraph neural networks. The structure of the hypergraph is another vital factor that affects the node classification accuracy of a hypergraph neural network. Thus, the questions of how to construct hypergraphs and how to define the relationship between nodes in the hypergraph are fundamental. Figure 1 shows a standard method for constructing hypergraphs that constructs hyperedges similar to the k-nearest neighbour (KNN) method, a commonly utilized method in citation networks, 3D visual object classification, and other applications. In this method, each node is chosen as the centre node, nodes with high correlations with the centre node are called neighbour nodes, and the centre node and neighbour nodes form a hyperedge. However, hypergraphs constructed in this manner cannot be used to describe the relationship between nodes in the same hyperedges. For example, suppose one node in a hyperedge is not in the same category as the central node and the other nodes are. In that case, the elimination of noise for nodes in different categories needs to be considered in the node classification task.

In this paper, we propose HGNN-AS, a hypergraph neural network that combines attention and self-attention mechanisms to improve the efficiency of HyperGAT by focusing on a hypergraph constructed in a KNN-like manner. In the self-attention mechanisms employed in the model, we leveraged nodes and hyperedges that encode explicit information about the importance of the relations in the hypergraph. If nodes i and j have a relationship between a central node and its corresponding neighbour node, they are more relevant to each other than to other nodes. If nodes i and j do not have this relationship, they are not important to each other. If node i is on hyperedge e, this relationship is relevant, and if node i is not on hyperedge e, this relationship has very little relevance. Although conventional attention is trained without self-attention, if we have prior knowledge, we can supervise attention using the self-attention mechanism. Specifically, we can use two new attention values as inputs to predict the relationship between the nodes and hyperedges. We combined these two self-attention mechanisms with HyperGAT, which slightly improved the performance of the model.

We also introduced a multihead attention mechanism, which has the same form as in the GAT, into our model. Specifically, our multihead attention mechanism adopts multiple independent attentional heads in the first-layer network to identify node features. It splices the obtained node features into the second-layer network to get the final results. The multihead attention method is an integration mechanism that can prevent overfitting and acquire multiple types of data features to increase accuracy.

Our contributions can be summarized as follows:

• We propose the HGNN-AS model for enhancing the accuracy of node classification tasks on hypergraphs.
• We design two self-attention mechanisms using the hyperedge information and node information to learn richer information and obtain better representations.
• We apply the multihead attention mechanism to hypergraphs, which increases the stability of the model and prevents overfitting.

This paper is organized as follows: In Section 2, we introduce related work, including graph neural networks, hypergraph neural networks, and self-attention mechanisms. In Section 3 we present our proposed HGNN-AS model as well as related details. In Section 4, the experiments are discussed and analysed. Section 5 concludes this paper.

2. Related Work

In this section, we briefly review the development history and existing work on graph neural networks, hypergraph neural networks, and self-attention mechanisms.

2.1. Graph Neural Networks

A graph is a simple, abstract, and intuitive mathematical expression between objects that is widely used in recommended systems, social networks, and other applications. With increasing data volume, traditional graph algorithms have considerable limitations in addressing critical issues, such as node classification and link prediction. The graph neural network (GNN) model considers the scale, heterogeneity, and deep topological information of the input data, among other factors, to achieve convincing and consistent results when mining deep practical topological information, extracting key complex data features, and rapidly processing large amounts of data. For example, GNNs have demonstrated broad potential across various application scenarios, including federated learning for text sentiment analysis [11], high-dimensional feature selection and classification optimization [12], dynamic task scheduling in edge computing [13], medical image segmentation and liver lesion analysis [14], vehicle routing optimization and dynamic delivery [15], IoT location privacy protection [16], data trading mechanism design [17], D-IHFSs-based uncertain information processing [18], and hesitant fuzzy linguistic term sets similarity combined with TOPSIS [19], as well as other specific applications [20,21,22].

In 2005, Gori et al. [23] introduced the concept of graph neural networks based on neural network research results and designed a model for processing data with graph structures. Scarselli et al. [24] extensively researched this model in 2009; since then, other graph neural network models and research applications have been proposed. With the increase in graph-structured data studies in recent years, the number of graph neural network research papers has increased dramatically, and the research direction and application domains of graph neural networks have grown substantially. In 2017, Kipf et al. [25] proposed graph convolutional networks (GCNs). In 2018, Velickovic et al. [7] proposed graph attention networks (GATs), which introduced the increasingly mature attention mechanism to natural language processing. GATs comprehensively capture the entity and relation features as well as semantically similar relationships in multihop domains by assigning different weights to various entities in the n-hop domain.

GCNs, GATs, and other conventional graph neural networks [26,27,28,29] can effectively extract structural and attribute information, providing convenient solutions to node classification tasks. However, due to graph structure limitations, these models can only represent second-order relations, impacting the acquired data to varying degrees. To better represent higher-order relationships between objects, researchers have extended these graphs to hypergraphs and have proposed hypergraph neural networks to extract higher-order correlation information between nodes.

2.2. Hypergraph Neural Networks

In contrast to graphs, which can model only second-order relationships among data, hypergraphs can represent higher-order relationships among data, allowing network models to extract more complete feature information and better learn graph representations. Hypergraphs have been widely applied in various fields, including spatiotemporal sequence analysis [30], recommendation systems [31], disease prediction [32], IoT anomaly detection and Edge AI [33], as well as air quality prediction frameworks [34].

The pioneering work on hypergraph learning [35] systematically elaborated the fundamental theory and properties of hypergraphs, laying the foundation for subsequent research in this area. Building upon this theoretical framework, Zhou et al. [36] were the first to investigate the use of hypergraphs to represent complex relationships among objects and extended spectral clustering methods to hypergraphs, further proposing additional hypergraph construction techniques based on spectral hypergraph clustering. Subsequently, Zhang et al. [37] proposed a dynamic hypergraph structure learning model, which enhances representational capacity by simultaneously optimizing the label projection matrix and hypergraph structure. Zhang et al. [38] introduced Hypergraph Label Propagation Networks (HLPNs), combining hypergraph-based label propagation with deep neural networks to improve feature embedding. In 2019, Feng et al. [4] proposed Hypergraph Neural Networks (HGNNs), which construct hypergraphs using the k-nearest neighbour (KNN) algorithm and achieved superior performance on classification tasks. To address HGNN’s limitation in dynamically adjusting the hypergraph structure via feature embeddings during training, Dong et al. [39] proposed the HNHN framework for hypergraph representation learning, wherein convolution operations over the incidence matrix update hypernode and hyperedge representations, enabling flexible weighting of high-cardinality hyperedges and high-degree nodes. Further exploring higher-order information extraction, Bai et al. [40] incorporated hypergraph convolution and hypergraph attention mechanisms into graph neural networks to capture higher-order information, demonstrating strong performance in semi-supervised node classification tasks.

Meanwhile, Duta et al. [41] introduced the concept of a cavity layer within hypergraphs, combining linear and nonlinear features to achieve more substantial inductive bias, surpassing traditional methods in node classification tasks. For complex structured data, Chen et al. [42] proposed the HyTrel framework, treating cells as nodes, rows, and columns and entire tables as different types of hyperedges, effectively capturing permutation invariance and hierarchical structure properties and achieving strong performance across multiple downstream tasks. Furthermore, to simultaneously model high-order dependencies among nodes and interactions among hyperedges while facilitating knowledge transfer, Ju et al. proposed the HEAL framework, introducing branch consistency constraints and achieving state-of-the-art performance on multiple real-world graph classification datasets [43].

2.3. Self-Attention Mechanism

The attention mechanism [44], which Treisman and Gelade proposed, is a model that simulates attention in the human brain. The attention mechanism is a signal connection mechanism modelled on observations in the human brain; that is, when humans scan an observation target, they focus on specific areas that contain key information while ignoring the remainder of the nonprimary information. The attention mechanism has been a popular method in various fields since it was first proposed. Bahdanau et al. [8] first applied the attention mechanism to machine translation. Rush et al. [45] applied the attention mechanism to text summaries and extracted keywords from long sentences and paragraphs. Mnih et al. [46,47] used an attention mechanism for image recognition, with substantial progress in the development of image captions. Chan [48] used an encoder–decoder framework based on the attention mechanism in speech recognition to establish the relationship between speech and words. Xu [49] introduced the attention mechanism to image captioning. The attention mechanism has also achieved good results in hypergraph learning. Kaize Ding et al. [6] proposed hypergraph attention networks (HyperGAT), which introduced the attention mechanism to hypergraph neural networks.

In 2017, the Google Translate team proposed the self-attention mechanism [9], another kind of attention mechanism. Compared with the conventional attention mechanism, the self-attention mechanism focuses on itself and extracts relevant information without using additional information. The content of the source and target ends differs in the traditional attention mechanism, and the outcome is the dependency between each word at the source and target ends. In the self-attention mechanism, attention occurs between source elements within the source or target. The self-attention mechanism has been used in a variety of graph neural network applications. For example, Kim [50] applied the self-attention mechanism to the graph attention network, which improved the effect of the original network. In recent years, the self-attention mechanism has been widely used for tasks in high-dimensional and complex scenarios, including user adoption behaviour analysis in mobile chronic disease management services, indoor wireless positioning optimization, big data service evaluation and optimization, as well as Edge server resource allocation, mobile charging, and tasks related to interval type-2 fuzzy sets (IT2 FS) [51,52,53,54,55].

3. Methodology

In this section, we first review HyperGAT and then review the details of our proposed HGNN-AS.

3.1. HyperGAT for Node Classification

With the increased focus on hypergraph learning and attention mechanisms, HyperGATs have experienced considerable success in representation learning on hypergraph-structured data. HyperGATs obtain new node representations by applying two neighbourhood aggregation functions on hypergraphs. A general HyperGAT layer can be defined as

$$\begin{array}{cc}\hfill f_j^l& =AGG{R}_{node}^l\left(\left\{h_k^{l-1}\right|\forall v_k\in e_j\}\right)\hfill \\ \hfill h_i^l& =AGG{R}_{edge}^l\left(h_i^{l-1}\left\{f_j^l\right|\forall e_j\in {\epsilon }_i\}\right)\hfill \end{array}$$

(1)

where $f_j^l$ is the representation of hyperedge $e_j$ in layer l, and $h_i^l$ is the node representation of node $v_i$ in layer l. $AGG{R}_{node}^l$ is an aggregation function that combines node features on hyperedges, and $AGG{R}_{edge}^l$ is another aggregation function that combines hyperedge features with nodes. ${\epsilon }_i$ denotes the set of hyperedges connected to node $v_i$.

HyperGAT aggregates node features and hyperedge features based on a dual attention mechanism. Although this model obtains a good node representation, the attention mechanism used in HyperGAT is insufficient. When the noise in different hypergraphs or irrelevant nodes is connected, the attention mechanism adopted by HyperGAT often fails to assign the correct weight to each node. Nodes should ignore the interference of different types of nodes as much as possible when aggregating neighbour nodes; however, with the original attention mechanism, the interference of different types of nodes is substantial. Another disadvantage of HyperGAT is that the meaning of the hyperedge representation is unknown when hyperedges aggregate node features with different weights to obtain the hyperedge representation. This representation is related to the construction of the hypergraphs. When the hypergraph is constructed like KNN, we believe that the representation of the hyperedge is the representation of the class closest to the central node. The second aggregation function $AGG{R}_{edge}^l$ of the HyperGAT model also expresses this potential meaning. When there is a significant difference between the node and hyperedge features, the attention mechanism adopted by the aggregation function assigns a small weight to this hyperedge. However, the attention mechanism in HyperGAT is insufficient and needs improvement. In summary, while the two aggregation functions of HyperGAT obtain exemplary node and hyperedge representations, the attention mechanism of HyperGAT can be improved.

3.2. Hypergraph Neural Network with Attention and Self-Attention

3.2.1. Data Preprocessing and Building Hypergraphs

The first step in the HGNN-AS model is constructing the hypergraphs. We construct a hypergraph like KNN, which selects each node as a centre node and chooses the nodes near the central node as its neighbour nodes. Then, each central node and its corresponding neighbour nodes form a hyperedge. A hypergraph is defined as $G=(V,E)$, where $V=\{v_1,\dots ,v_n\}$ represents all nodes in the hypergraph, and $E=\{e_1,\dots ,e_m\}$ represents all hyperedges in the hypergraph. It is worth noting that each hyperedge e could have two or more nodes. Notably, an incidence matrix $A\in R^{n\times m}$ can also be used to depict the topological structure of a hypergraph G. This matrix is defined as follows:

$$A(v,e)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}v\in e\hfill \\ 0,\hfill & \mathrm{if}v\notin e\hfill \end{array}\right.$$

(2)

Typically, each node in the hypergraph has a d-dimensional attribute vector. Therefore, the attribute vectors of all nodes can be represented as $X={[X_1,X_2,X_3,\dots ,X_n]}^T$∈ $R^{n\times m}$, and we can represent the whole hypergraph by $G=(A,X)$.

The second step in the HGNN-AS model is adding two new labels according to the hypergraph structure. The first label assigns node pairs in which the centre node $v_i$ is connected to the neighbour node a value of 1, and node pairs without this connection have a value of 0. The first label can be defined as $labe{l}_1=[l{1}_{11},l{1}_{12},\dots ,l{1}_{ij},\dots ,l{1}_{nn}]$. The second label assigns nodes that exist on the hyperedge in the graph a value of 1 and nodes that do not exist on the hyperedge a value of 0. Thus, the second label can be defined as $labe{l}_2=[l{2}_{11},l{2}_{12},\dots ,l{2}_{ij},\dots ,l{2}_{nm}]$. These two labels can be defined as follows:   

$$\begin{array}{cc}\hfill labe{l}_1& =[l{1}_{11},l{1}_{12},\dots ,l{1}_{ij},\dots ,l{1}_{nn}]\hfill \\ \hfill A(v,e)& =\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{centre}\mathrm{node}{v}_i\mathrm{is}\mathrm{connected}\mathrm{to}\mathrm{neighbour}\mathrm{node}{v}_j\hfill \\ 0,\hfill & \mathrm{if}\mathrm{centre}\mathrm{node}{v}_i\mathrm{is}\mathrm{not}\mathrm{connected}\mathrm{to}\mathrm{neighbour}\mathrm{node}{v}_j\hfill \end{array}\right.\hfill \\ \hfill labe{l}_2& =[l{2}_{11},l{2}_{12},\dots ,l{2}_{ij},\dots ,l{2}_{nm}]\hfill \\ \hfill A(v,e)& =\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{v}_{i}on{e}_j\hfill \\ 0,\hfill & \mathrm{if}{v}_{i}isnoton{e}_j\hfill \end{array}\right.\hfill \end{array}$$

(3)

3.2.2. Node-Level Attention

The whole process of the HGNN-AS model is shown in Figure 2. The attention mechanism used in the HGNN-AS model is the same as that used in HyperGAT. This mechanism first aggregates node features and then aggregates hyperedge features. The specific process of node-level attention is as follows:

$$f_j^l=\sigma \left({\alpha }_{jk}{w}_1{h}_k^{l-1}\right)$$

(4)

where $\sigma$ represents the activation function, for example, $RELU$, $w_1$ is a trainable weight matrix, and $h_k^{l-1}$ represents the l-1th layer node $v_k$. The attention coefficient of node $v_k$ in hyperedge $e_j$ is denoted by ${\alpha }_{jk}$ and can be calculated as follows:

$$\begin{array}{c}\hfill {\alpha }_{jk}={\displaystyle \frac{exp\left(a_1^T{u}_k\right)}{{\sum }_{v_p}\in e_{j}exp\left(a_1^T{u}_p\right)}}\\ \hfill u_k=LeakeyReLU\left(w_1{h}_k^{l-1}\right)\end{array}$$

(5)

$v_k$ where $a_1^T$ is a trainable weight vector.

3.2.3. Node and Node-Level Self-Attention

We determined the hyperedge representations through the above node-level attention process. We added a self-attention mechanism to determine whether nodes were connected to each other in the final layer of the network, which can be used to learn better hyperedge representations. This self-attention mechanism can be defined as follows:

$$S_{NN}=\sigma \{[{\left(w_1{h}_i^{l-1}\right)}^T·\left(w_1{h}_j^{l-1}\right)]·(a_1^T{u}_i+a_1^T{u}_j)\}$$

(6)

We assume that l is the last layer of the HGNN-AS model. $h_i^{l-1}$ and $h_j^{l-1}$ represent the node representations of nodes $v_i$ and $v_j$ in the $l-1$ layer, respectively. The degree of association between nodes $v_j$ and $v_j$ is represented by $S_{NN}$. $\sigma$ represents the sigmoid activation function. Our self-attention mechanism has the same parameters as the original HyperGAT self-attention mechanism. The goal of the self-attention mechanism at the node level is to better train the weight matrix $w_1$ and weight vector $a_1^T$. This goal was motivated by SuperGAT [6] and the gating mechanism of gated recurrent units [56].

3.2.4. Edge-Level Attention

All hyperedge representations $\left\{f_j^l\right|\forall e_j\in {\epsilon }_i\}$ were obtained through node-level attention. To learn the next-layer representation of node $v_i$, we use an edge-level attention mechanism to identify informative hyperedges. This procedure can be formalized as follows:

$$h_i^l=\sigma \left(\sum _{v_k\in e_j}{\beta }_{ij}{w}_2{f}_j^l\right)$$

(7)

where $h_i^l$ is the output representation of node $v_i$, and $w_2$ is a weight matrix. The attention coefficient of hyperedge $e_j$ on node $v_i$ is denoted by ${\beta }_{ij}$ and can be calculated as follows:

$$\begin{array}{cc}\hfill {\beta }_{ij}& ={\displaystyle \frac{exp\left(a_2^T{u}_j\right)}{{\sum }_{v_p\in {\epsilon }_i}exp\left(a_2^T{v}_p\right)}}\hfill \\ \hfill u_j& =LeakeyReLU\left([w_2{f}_j^l\parallel w_1{h}_i^{l-1}]\right)\hfill \end{array}$$

(8)

where $a_2^T$ is a weight vector used to calculate the importance of the hyperedges, and $\parallel$ is the concatenation operation.

3.2.5. Node and Hyperedge-Level Self-Attention

The second self-attention mechanism we introduce is called node and hyperedge level self-attention, which judges whether a node is on a hyperedge based on the hyperedge and node representations. The process can be formulated as follows:

$$S_{NE}=\sigma \{[{\left(w_2{f}_j^l\right)}^T·\left(w_1{h}_i^{l-1}\right)]·(a_2^T\left[w_2{f}_j^l\parallel w_1{h}_i^{l-1}]\right)\}$$

(9)

We assume that l is the last layer of the neural network and that $f_j^l$ is the representation of the hyperedge $e_j$ in the lth layer. The representation of node $v_i$ in the $l-1$ layer is represented by $h_i(l-1)$. $S_{NE}$ is the degree of association between the node and the hyperedge, and $\sigma$ represents the sigmoid activation function. The two self-attention mechanisms are shown in Figure 3. Compared with the original hypergraph attention network, our new self-attention mechanism does not add any new parameters. The goal of the self-attention mechanism at the node and hyperedge levels is to better train the weight matrix $w_2$ and weight vector $a_2^T$ to allow our model to learn better node representations. Algorithm 1 shows the pseudocode that summarizes the training procedure.

Algorithm 1 Training process of the HGNN-AS model.

Require: The hypergraph $G=(V,E)$,                 The node feature {$h_i$,$\forall i\in V$ },                 The number of attention head K, Ensure: The final nodes representation {$h_i^2$,$\forall i\in V$ },                   The degree of association between nodes $S_{NN}$,                   The degree of association between nodes and hyperedges $S_{NE}$,        initialization        First layer: Aggregate features using a multihead attention mechanism                 for $K=1\dots K$ do                          Aggregate node features to hyperedges $f_j^1$ ←$AGG{R}_{node}\left(h_i\right)$                          Aggregate hyperedge features to nodes $h_i^1$ ←$AGG{R}_{edge}(f_j^1,h_i)$                 end        Second layer: Aggregate features using the self-attention mechanism                 Aggregate node features to hyperedges $f_j^2$ ←$AGG{R}_{node}\left(h_i^1\right)$                 $S_{NN}$ ← Node and node-level self-attention                 Aggregate hyperedge features to nodes $h_i^2$ ←$AGG{R}_{edge}(f_j^2,h_i^1)$                 $S_{NE}$ ← Node and edge-level self-attention        return $h_i^2$,$S_{NN}$, $S_{NE}$

3.2.6. Loss Function

Finally, we combine the cross-entropy loss of the node labels ($L_O$), the node and node-level self-attention loss of the final layer ($L_{EE}$), and the node and hyperedge level self-attention loss of the final layer ($L_{NE}$) to establish the total loss function, which can be defined as follows:

$$\begin{array}{cc}\hfill L& =L_O+L_{EE}+L_{NE}\hfill \\ \hfill L_{EE}& =BCEWithLogitsLoss(Labe{l}_1,S_{EE})\hfill \\ \hfill L_{NE}& =BCEWithLogitsLoss(Labe{l}_2,S_{NE})\hfill \end{array}$$

(10)

For $L_{EE}$ and $L_{NE}$, we use the $BCEWithLogitsLoss$ loss function. Algorithm 1 reports the pseudocode that summarizes the training procedure.

4. Experiments

In this section, we test our proposed HGNN-AS model on two tasks: citation network classification and visual object recognition. In addition, we compare the proposed method to HyperGAT and other advanced methods.

4.1. Citation Network Classification

Datasets

In this experiment, we classify a citation network dataset. We employ two widely utilized citation network datasets, namely Cora and Pubmed [57]. The experimental design was based on semi-supervised learning with graph embeddings [58] proposed by Yang et al. The nodes in both datasets represent papers, and the edges represent the citation relationships between papers. The node features of the Cora dataset have 1433 dimensions, and the nodes are divided into 7 categories. The node features of the Pubmed dataset have 500 dimensions, and the nodes are divided into 3 categories. We used semi-supervised learning to classify the nodes, and 140/500/1000 nodes in the Cora dataset were used for training/validation/testing. Additionally, 60/500/1000 nodes in the Pubmed dataset were used for training/validation/testing. Specific descriptions of the two datasets are listed in

Table 1. Summary statistics of the two citation network datasets.

Dataset	Cora	Pubmed
Node	2708	19,717
Edge	5429	44,338
Feature	1433	500
Class	7	3

.

To create the hypergraph structure for HGNN-AS, one node in the graph was chosen as the centre node, and all nodes connected to this central node were selected as neighbour nodes. We generated the hyperedge between the central node and its corresponding neighbour nodes. We constructed a hypergraph according to these nodes and hyperedges that has the same amount of data as the original graph. Our constructed hypergraph has the same structure as the original graph; however, the view of the graph is different.

4.2. Experimental Settings

A two-layer HGNN-AS was used in this experiment. The first layer had $K=4$ attention heads, each of which computed $F=16$ features (for a total of 64 features), and dropout [59] with a drop rate of $p=0.5$ was used to prevent overfitting. ReLU was chosen as the nonlinear activation function. The cross-entropy loss function was used as the original HyperGAT loss function, and the BCEWithLogitsLoss function was used as the self-attention loss function. We used the Adam optimizer [57] with a learning rate of 0.005 during the training process to minimize the cross-entropy loss. In these experiments, we also compared the proposed HGNN-AS to existing methods.

4.3. Results and Discussion

Table 2. Test accuracy (%) with 95% confidence intervals (CIs) on citation network datasets.

Method	Cora Acc (%)	95% CI	Pubmed Acc (%)	95% CI
GCN [25]	81.5	81.0–82.0	79.0	78.5–79.5
GAT [10]	83.0	82.5–83.5	79.0	78.5–79.5
HGNN [4]	81.6	81.2–82.0	80.1	79.8–80.4
HyperGAT [58]	82.9	82.4–83.4	80.0	79.6–80.4
HGNN-AS	83.9	83.2–84.3	80.3	80.0–80.6

displays the experimental results and comparisons on the citation network datasets. On the Cora and Pubmed datasets, the proposed HGNN-AS model achieved average classification accuracies of 83.9% and 80.3%, respectively, over 200 independent runs. To more intuitively assess the reliability of the improvement, we calculated the corresponding 95% confidence intervals (CIs) for the runtime performance. The results indicate that HGNN-AS outperforms or performs comparably to other state-of-the-art methods. Compared with HyperGAT, HGNN-AS achieves a slight improvement on the Pubmed dataset and approximately a 1.0% gain on the Cora dataset. In future work, we plan to enhance model performance further and validate the statistical significance of the improvements through more robust hypergraph construction and noise suppression strategies. The performance of the model under different dropout rates on the Cora and Pubmed datasets is shown in Figure 4. It is noteworthy that HGNN-AS exhibits greater stability than HyperGAT as the dropout rate varies, suggesting that HGNN-AS is less prone to overfitting. In addition, we compared the loss curves of HGNN-AS with those of HyperGAT and HGNN on both datasets, as shown in Figure 5. It can be observed that HGNN-AS achieves the fastest convergence speed and the lowest final training loss on both datasets, demonstrating that the self-attention mechanism effectively enhances the model’s learning efficiency and training stability.

4.4. Visual Object Classification

4.4.1. Datasets

In this experiment, we classified visual objects. To evaluate our methods, we used two public benchmark datasets: the Princeton ModelNet40 dataset [60] and the National Taiwan University (NTU) 3D model dataset [61]. We follow HGNN and preprocessed the data with multiview convolutional neural networks (MVCNNs) [62] and group-view convolutional neural networks (GVCNNs) [63]. The ModelNet40 dataset contains 12,311 objects in 40 popular categories, and the same training/testing split is used as in [60], with 9843 objects used for training and 2468 objects used for testing. The NTU dataset contains 2012 3D shapes in 67 different categories, such as cars, chairs, chess, chips, clocks, cups, doors, frames, pens, and plant leaves. In the NTU dataset, 80% of the data was used for training, while the remaining 20% of the data was used for testing. Each 3D object in both datasets is represented by an extracted feature. To extract the features of each 3D object, we used two methods: the MVCNN and the GVCNN. Specific descriptions of the two datasets are listed in

Table 3. Detailed information on the ModelNet40 and NTU datasets.

Dataset	ModelNet40	NTU
Objects	12,311	2012
MVCNN Feature	4096	4096
GVCNN Feature	2048	2048
Training node	9843	1639
Testing node	2468	373
Classes	40	67

.

Therefore, each 3D object in the ModelNet40 and NTU datasets is composed of an MVCNN feature and a GVCNN feature, as well as its own label. Compared to the HyperGAT method, the ModelNet40 and NTU datasets have no available graph structures. As a result, we constructed probability graphs based on the node distance, which is the same method used in HGNN models.

4.4.2. Experimental Settings

We used the ModelNet40 and NTU datasets in this experiment. To construct the hypergraph, we used 10 nodes with the shortest Euclidean distances from the centre node as neighbour nodes, and we used the hypergraph and object features as inputs. We combined the MVCNN and GVCNN features in different ways to obtain the results of our model when processing single modality features and multimodality feature. The data were preprocessed according to HGNN.

In this experiment, we used a two-layer hypergraph attention network with a self-attention mechanism. As shown in

Table 4. Parameter settings on the ModelNet40 and NTU datasets.

Experimental Parameter	Setting
Attention head	2
Feature dimension	128
Optimizer	Adam optimizer
Learning rate	0.001
Dropout rate	0.3
Epochs	1000

, the first layer has two attention heads, and the hidden layer obtained by each attention head has a feature dimension of 128 (for a total of 256 features). For optimal performance, we used the Adam optimizer and the optimized learning rate was set to 0.001, and the dropout rate was set to 0.3. We trained the model for 1000 epochs with an early-stopping strategy to learn the HGNN-AS model. The cross-entropy loss function was used as the node classification loss function, and the BCEWithLogitsLoss function was used as the loss function of the self-attention mechanism.

4.4.3. Results and Discussion

Figure 6 illustrates the feature aggregation behaviour of the HGNN-AS model in modelling node–hyperedge relationships. We visualised the attention weights for samples from the ModelNet40 dataset. It can be clearly observed that, in the first layer, the attention distribution is relatively dispersed, with the model tending to aggregate features from a broader neighbourhood to capture local structural information. In the second layer, the attention gradually focuses on a small number of key nodes and hyperedges, indicating that the model begins to emphasise essential structures. This transition from dispersed to concentrated attention suggests that HGNN-AS can effectively identify critical structures, thereby supporting subsequent feature aggregation and enhancing both the interpretability and robustness of the model.

The experimental results for two visual object classification tasks are shown in

Table 5. Test accuracy (%) on visual object classification. BOTH means GVCNN + MVCNN, which stands for combining features or structures to generate multiheadal data.

Datasets	Feature	Structure	GCN [25]	HGNN [4]	HyperGAT [58]	HGNN-AS
NTU	MVCNN	MVCNN	86.7%	91.0%	90.2%	91.1%
	GVCNN	GVCNN	91.8%	92.6%	92.4%	93.0%
	MVCNN	BOTH	92.3%	96.6%	96.4%	96.7%
	GVCNN	BOTH	92.8%	96.6%	96.7%	97.0%
	BOTH	BOTH	94.4%	96.7%	96.7%	97.0%
Model-Net40	MVCNN	MVCNN	71.3%	75.6%	76.9%	77.4%
	GVCNN	GVCNN	78.8%	82.5%	82.3%	82.6%
	MVCNN	BOTH	73.2%	83.6%	82.8%	83.7%
	GVCNN	BOTH	75.9%	84.2%	83.9%	84.2%
	BOTH	BOTH	76.1%	84.2%	84.0%	84.2%

. We find that HGNN-AS outperformed HyperGAT in all cases on these two datasets for both single modality and multimodality features. More specifically, our method has improvements ranging from 0.3% to 0.9% on different datasets. Compared with general graph neural networks, when only one feature is used to generate graph/hypergraph structures, HGNN-AS can achieve a certain level of improvement. However, when we use both MVCNN and GVCNN features to construct a hypergraph, HGNN-AS achieves much better performance compared with GCN. These results reveal that HGNN-AS is a hypergraph neural network with good performance, which is more suitable for processing multiheadal data.

5. Conclusions

In this paper, we propose a hypergraph neural network for node classification that combines attention and self-attention mechanisms (HGNN-AS), which is an improvement on the HyperGAT network. In this method, the HGNN-AS model is generated by adding a multihead attention mechanism and two self-attention mechanisms to the HyperGAT model. These two new self-attention mechanisms can obtain better node representations and hyperedge representations, and the multihead attention mechanism stabilises the training effect. To the best of our knowledge, this is the first multihead attention mechanism that has been added to hypergraph neural networks. Our proposed HGNN-AS network targets hypergraphs constructed like KNN, which are more accurate for node classification tasks.

Although HGNN-AS has achieved considerable progress, some limitations remain that should be addressed in the future. (1) Although we provided a new hypergraph neural network model, our hypergraph attention and hypergraph structure analyses are still unclear. (2) Due to the similarities between hypergraphs and heterogeneous graphs in processing multimodal data, we can use the self-attention mechanism to learn richer information in heterogeneous graphs.