Learning Heterogeneous Graph Embedding with Metapath-Based Aggregation for Link Prediction

Abstract

Along with the growth of graph neural networks (GNNs), many researchers have adopted metapath-based GNNs to handle complex heterogeneous graph embedding. The conventional definition of a metapath only distinguishes whether there is a connection between nodes in the network schema, where the type of edge is ignored. This leads to inaccurate node representation and subsequently results in suboptimal prediction performance. In heterogeneous graphs, a node can be connected by multiple types of edges. In fact, each type of edge represents one kind of scene. The intuition is that if the embedding of nodes is trained under different scenes, the complete representation of nodes can be obtained by organically combining them. In this paper, we propose a novel definition of a metapath whereby the edge type, i.e., the relation between nodes, is integrated into it. A heterogeneous graph can be considered as the compound of multiple relation subgraphs from the view of a novel metapath. In different subgraphs, the embeddings of a node are separately trained by encoding and aggregating the neighbors of the intrapaths, which are the instance levels of a novel metapath. Then, the final embedding of the node is obtained by the use of the attention mechanism which aggregates nodes from the interpaths, which is the semantic level of the novel metapaths. Link prediction is a downstream task by which to evaluate the effectiveness of the learned embeddings. We conduct extensive experiments on three real-world heterogeneous graph datasets for link prediction. The empirical results show that the proposed model outperforms the state-of-the-art baselines; in particular, when comparing it to the best baseline, the F1 metric is increased by 10.35% over an Alibaba dataset.

1. Introduction

With the coming of the era of science and technology, various network applications make people’s lives more convenient. However, the diversity of items leads to the problem of information overload. A promising solution is to learn the embeddings of users and items, which can be applied to various downstream tasks, such as link prediction [1,2] and recommendation systems [3]. As deep learning has developed, graph embedding, which can map nodes in a graph to low-dimensional continuous spaces, has been proposed and widely used because many natural representations of real-world datasets are made from graph data structures. DeepWalk [4], LINE [5], and node2vec [6] are pioneering works to learn node embeddings. DeepWalk and node2vec feed node sequences to a skip-gram model to learn the embeddings of nodes. The sequences are generated by random walks in the model. LINE studies the problem of embedding very large information networks into low-dimensional vector spaces by exploiting the first-order and second-order proximity between nodes.

Recently, graph neural networks (GNNs), the new products of graph-embedding models, have attracted a great deal of attention. GNNs, such as the graph convolutional network (GCN) [7], GraphSAGE [8], the graph attention network (GAT) [9], and many other variants [10,11,12,13], have proven to be strongly expressive and demonstrate state-of-the-art performance on various graph-based downstream tasks, such as link prediction [1,2], community detection [14,15], node classification [16,17], and the recommendation system [3]. All these GNN-based approachs can effectively reserve network structures and inherent properties with the specific message propagation.

Nevertheless, most methods were designed to resolve only the homogeneous graph, i.e., there is only one type of node and edge. In real life, graph data usually consists of different types of nodes and edges. For example, an e-commerce dataset includes two types of nodes at least, namely users and items. Furthermore, there are various types of edges. For example, clicking, add to favorite, add to cart, and purchase are classic behaviors in the e-commerce scenario. Each kind of behavior denotes one type of edge. We refer to this kind of graph as heterogeneous information networks or heterogeneous graphs. In heterogeneous graphs, traditional GNNs cannot encode the rich structural and semantic information into a low-dimensional vector space because they treat all nodes equally [18].

Many researchers have proposed graph-embedding methods based on the idea of metapaths [19] to deal with heterogeneous graphs [20,21,22]. In particular, a metapath is an ordered and composite relation sequence connecting different or identical node types. For example, an e-commerce dataset includes two node types, i.e., user and item, and various edge types include clicking, add to favorite, add to cart, and purchase. $User$$\stackrel{Add-to-cart}{\to }$$Item$$\stackrel{Clickedby}{\to }$$User$ is a metapath which describes a composite relation between users. With the aid of various metapath instances, each node can be directly connected with its high-order neighbors, and the high-order proximity between two nodes is hidden in heterogeneous graphs [18].

Although the metapath-based approaches have had some success in heterogeneous graph embedding, they still have the following limitations. (1) The conventional definition of metapath [23] only distinguishes whether there is a connection between two nodes in the network schema; the type of edge is ignored. This leads to inaccurate node-representation learning and subsequently leads to the suboptimal prediction performance. In heterogeneous graphs, a node may be connected by multiple types of edges. In fact, each type of edge represents one kind of scene. The intuition is that if the embedding of nodes are trained under different scenes, the complete representation of nodes can be obtained by organically combining them. (2) In the aspect of node aggregation, classical network models are used, such as GCN [7] and graphSAGE [8]. GCN uses the adjacency matrix and degree matrix for aggregation operation. Although the adjacency matrix and degree matrix are obtained on the whole graph, they are suitable for operating on the whole graph and not for aggregating on the metapath. With the increasing of metapath length, the traditional aggregation methods (e.g., mean, maximum) aggregate all the node information with the same weight while ignoring their important differences, which leads to the deviation of node representation.

To solve the problems of multiple types of edge relations and aggregation over the metapath instances, we first propose a novel definition of metapath in which the edge type is considered. A heterogeneous graph can be considered as the compound of multiple relation subgraphs from the view of novel metapath. In different subgraphs, the embeddings of a node are separately trained by encoding and aggregating the neighbors of the intrapaths, which are the instance level of novel metapaths. Then, the final embedding of the node is obtained by use of the attention mechanism which aggregates from the interpath, which is the semantic level of novel metapaths. Finally, we make extensive experiments on three real-world heterogeneous graph datasets for link prediction. The empirical results demonstrate that our model is superior to the state-of-the-art baselines.

In summary, our contributions can be summarized in the following three points.

•

Novel definition: We propose a novel definition of metapath, i.e., relation-constrained metapath. Then, each relation constrained metapath instance has a specific meaning and can clearly describe the relations between nodes.

•

Multiview modeling: We design two levels of metapath-guided aggregation methods, which aggregates from the intrapath on the instance level and the interpath on the semantic level, respectively. This method is more conducive to node aggregation and improves the accuracy of nodes representations.

•

Multifaceted experiments: We perform a suite of experiments on three datasets. The results demonstrate the effectiveness of the proposed model compared to the baseline model. In addition, we validate that our model can alleviate cold-start problem to a certain extent.

The remainder of this paper is organized as shown below. In Section 2, we review the related work. We introduce some definitions in heterogeneous graph and summarize the notations used in Section 3. In Section 4, we present our proposed model. In Section 5, a number of experiments of the proposed model are conducted and detailed analyses are illustrated. The model and future research directions are summarized in Section 6.

2. Related Work

Graph embedding. The goal of graph embedding is to project nodes into a low-dimensional vector space, in which the embedding can be used for many downstream tasks, e.g., node classification, graph classification, community detection, and link prediction. DeepWalk [4] and node2vec [6] generate corpus on graph by random work and then feed corpus to a skip-gram model to learn the embeddings of nodes. LINE [5] studies the problem of embedding large-scale networks into low-dimensional space of vector while preserving the first-order and second-order proximity between nodes. Node2vec [6] devises a biased random walk programming to effectively exploit diverse neighbourhoods. Along with the rapid development of deep learning, many researchers propose GNNs to explore graph embedding because each node is naturally determined by its own attributes and its neighbours. GraphSAGE [8] proposes a layer-by-layer aggregation of attributes of neighbours while implicitly exploring the topological structure. Inspired by the Transformer [24], GAT [9] introduces the graph-attention mechanism to measure the relative contributions of the neighbors in aggregation from the perspective of target nodes. DH-HGCN [3] models the dual homogeneity from social relations and item connections by hypergraph convolution networks to obtain high-order correlations among users and items. These methods have made considerable advancements in a variety of tasks with homogeneous graphs or specific graph structures. However, when applied to heterogeneous graphs, they are unable to encode heterogeneity into the representation.

Heterogeneous graph embedding. A heterogeneous graph involves multiple types of nodes and multiple types of edges, and it can depict the real world more truly. Heterogeneous graphs can describe much more complex relationships and structures than homogeneous graphs, so they have been widely used in graph data mining. Typical heterogeneous graphs include social networks, e-commerce systems, knowledge graphs, and citation networks. A variety of heterogeneous graph-representation learning methods have been proposed. They can be roughly divided into two categories. One involves graph-based approaches, such as HetGNN [25] and HAN [26]. HetGNN [25] jointly considers node-heterogeneous content encoding, type-based neighbour aggregation, and heterogeneous-type combinations. HAN [26] proposes a heterogeneous graph neural network based on the hierarchical attention. The other available method for heterogeneous graphs embedding involves metapath-based methodologies [18,19]. Metapath2vec [23] formalizes metapath-based random walks to construct the heterogeneous neighbourhood of a node and then leverages a heterogeneous skip-gram model to perform node embeddings. GATNE [27] formalizes the problem of graph embeddings of multiplex heterogeneity and proposes to solve this problem with transitional and inductive settings. MAGNN [18] applies three building block components, which settles the three characteristic limitations of current heterogeneous graph-embedding methods, and performs state-of-the-art performance on three real-world datasets with regard to the tasks of node clustering, node classification, and link prediction. Ref. [22] extracts multifaceted meaningful semantics reflected by the metapath on the heterogeneous graph as multiviews for both users and items, and effectively enhances user/item relationships on different aspects. HeCo [21] proposes two views of a heterogeneous graph (network schema and metapath views) to learn node embeddings in order to capture both local and higher-order structures simultaneously. MG-PFCM [20] defines the user-oriented and item-oriented metapaths, and performs the metapath-guided heterogeneous graph learning to improve the user and item embeddings. Although metapaths can deliver semantic information in heterogeneous graphs, it is too rough to capture the subtle semantics of certain applications. More specifically, the conventional definition of metapath is limited to whether there is a connection between nodes in the network schema. However, there are usually multiple types of edges between two nodes in heterogeneous graphs. The conventional definition of a metapath cannot capture the subtle semantics. In this paper, a type of edge is called one scene. A node has different representations in each scene. The intuition is that if the embedding of a node is trained under different scenes, the complete representation of the node can be obtained by organically combining them. Aiming at this problem, we propose a novel relation-constrained metapath.

Link prediction in heterogeneous graph. Inferring missing links between nodes in a graph based on observed interactions is known as link prediction [28,29,30], which is useful in numerous domains of application. Examples include bioinformatics [31], social media [32], e-commerce [33], and collaboration networks [34]. Many studies have been developed for link prediction in heterogeneous graphs. Ref. [35] proposes the problem of link prediction in heterogeneous networks as a multitask, metric learning problem. Ref. [30] considers the multinetwork scenario to encode diverse network structures of anchor users. Ref. [36] iteratively calculates link likelihoods taking longer paths between nodes into account. Ref. [37] first calculates node and edge relevance based on the summarized graph, and then combines both these factors to perform link prediction on unconnected pairs of nodes. Ref. [38] provides a systematic and comprehensive survey on hyperlink prediction. Link prediction is a downstream task to evaluate the effectiveness of the learned embeddings. We adopt it to evaluate effectiveness of the proposed method.

3. Preliminaries

3.1. Definitions

Definition 1.

Homogeneous and heterogeneous graph. Given a graph $G=(\mathcal{V},\mathcal{E})$, $\mathcal{V}$ and $\mathcal{E}$ represent the set of nodes and edges, respectively. The associated node and edge type mapping function are defined as ϕ: $\mathcal{V}\to \mathcal{T}$ and ψ: $\mathcal{E}\to \mathcal{R}$, respectively. Each node $v\in \mathcal{V}$ belongs to a particular node type $\varphi \left(v\right)\in \mathcal{T}$, and each edge $e\in \mathcal{E}$ belongs to a particular edge type $\psi \left(e\right)\in \mathcal{R}$. When $\left|\mathcal{T}\right|+\left|\mathcal{R}\right|>2$, the graph is called a heterogeneous graph; otherwise, it is a homogeneous graph.

In a heterogeneous graph, there are multiple node types and edge types. For example, in the e-commence datasets, they can be depicted by a heterogeneous graph in which two node types, i.e., users and items, and four edge types, i.e., clicking, add to favorite, add to cart, and purchase, are usually included.

Definition 2.

Metapath. In a heterogeneous graph, a metapath is an ordered and composite relation sequence connecting different or identical node types. A metapath $\mathcal{P}$ is denoted in the form of

$${\mathcal{T}}_1\stackrel{{\mathcal{R}}_1}{\to }{\mathcal{T}}_2\stackrel{{\mathcal{R}}_2}{\to }\dots \stackrel{{\mathcal{R}}_l}{\to }{\mathcal{T}}_{l+1},$$

(1)

where l is the length of $\mathcal{P}$.

For example, in the e-commence datasets, metapath

$${\mathcal{P}}_1:User\stackrel{Purchase}{\to }Item\stackrel{Purchasedby}{\to }User$$

(2)

can describe the copurchase relation. Another metapath,

$${\mathcal{P}}_2:User\stackrel{Add-to-cart}{\to }Item\stackrel{Clickby}{\to }User\stackrel{Purchase}{\to }Item,$$

(3)

describes a composition operator on relations between users and items.

A path $p:v_1\to v_2\to \dots \to v_{l+1}$ between nodes $v_1$ and $v_{l+1}$ is called a metapath instance of $\mathcal{P}$, if $\forall i$, the node $v_i$ and edge $e_i=(v_i,v_{i+1})$ satisfy $\varphi \left(v_i\right)\in \mathcal{T}$ and $\psi \left(e_i\right)\in \mathcal{R}$. For example, in the e-commence datasets, the path

$$u_1\stackrel{Purchase}{\to }{i}_2\stackrel{Purchasedby}{\to }{u}_3$$

(4)

and

$$u_1\stackrel{Add-to-cart}{\to }{i}_4\stackrel{Clickby}{\to }{u}_5\stackrel{Purchase}{\to }{i}_8$$

(5)

are path instances of ${\mathcal{P}}_1$ and ${\mathcal{P}}_2$, respectively.

The metapaths and instances indicate that there are potential relationships between nodes. However, in the heterogeneous graph, the various edge types make the composition relations between nodes quite complicated. It is difficult to study the nodes’ representations by use of the conventional metapath. Consequently, we propose a novel definition of metapath in which the edge type is integrated into it. The novel metapath is called a relation-constrained metapath, which can be formalized as shown below.

Definition 3.

Relation-constrained metapath. A relation-constrained metapath is a metapath based on a certain constraint that is denoted as ${\mathcal{P}}^{\mathcal{C}}$, where $\mathcal{P}$ is a metapath, and $\mathcal{C}$ represents the constraint on the relations in the metapath.

Note that the $\mathcal{C}$ can only be one constraint condition on a metapath. For example, the metapath ${\mathcal{P}}_1$ is equivalent to a relation-constrained metapath ${\mathcal{P}}_1^{\mathcal{C}=“purchase”}$: $User-Item-User$, abbreviated as ${\mathcal{P}}_1^{\mathcal{C}=“purchase”}$: $U-I-U$ or $UIU$. Each relation-constrained metapath instance has a specific meaning and can clearly describe the relations between nodes.

3.2. Problem Definition

Given a heterogeneous graph $G=(\mathcal{V},\mathcal{E})$ and a set of relation-constrained metapaths S, the problem of heterogeneous graph embedding is mapping each node v to a low-dimensional space representation, i.e., the goal is to find a function $fun:v\to {\mathbb{R}}^d$ for every node v, where $d\ll \left|\mathcal{V}\right|$. We adopt link prediction, which infers missing links between nodes, as the downstream task to evaluate the effectiveness of the learned embeddings.

3.3. Notations

The notations we will use throughout the paper are summarized in

Table 1. Notations used in the paper.

Notations	Definitions
$U,I$	User type, item type
$u,i$	A user $u\in U$, an item $i\in I$
$\mathcal{V},\mathcal{E}$	Node set, edge set
S	A set of relation-constrained metapath
v	A node $v\in \mathcal{V}$
d	The dimension of feature representation of nodes
$\overrightarrow{h_i}$	The feature representation of node $v_i$
$\mathcal{T},\mathcal{R}$	Node type set, edge type set
$\mathcal{P},l$	A metapath, the length of metapath
${\mathcal{P}}^{\mathcal{C}}$	A relation-constrained metapath
$\mathbf{A},{\mathbf{A}}^{\prime }$	The matrix including all nodes features in the sequence, the matrix after linear transformation
$Q,K,V$	The matrix representation after linear transformation
$l^{\prime }$	The length of sequence
L	The number of negative samples per training sample

.

4. The Proposed Method

In the following section, we take the process of the vector representation of $u_1$ as an example to present the proposed model in detail. As shown in Figure 1, the model starts with the instantiation process of metapaths, followed by the two-level aggregation. Then, we provide the optimization method and pseudocode of the proposed method. Finally, the classifier for link prediction is introduced.

4.1. Instantiating Paths and Generating Neighbors

4.1.1. Random Walk-Based Metapath Instance and Neighbour Generation

In heterogeneous graph, each edge type denotes a scene. The intuition is that the optimal vector representation of a node can be obtained by organically combining the various scene’s vector representations of it. First, the heterogeneous graph is decomposed into multiple relation subgraphs according to the user’s behaviors. Then, random walk is used for these relation subgraphs. Next, the sequences of nodes are obtained. Finally, top-k neighbors are generated depending on the frequency wherein the nodes appear in the sequence.

4.1.2. Relation-Constrained Metapath Instance and Neighbor Generation

As shown in Figure 1, there are two relation-constrained metapaths: ${\mathcal{P}}_1^{\mathcal{C}=“purchase”}$: $U-I-U$ and ${\mathcal{P}}_1^{\mathcal{C}=“add-to-cart”}$: $U-I-U$, which correspond to different user behaviors. The instantiation process of the relation-constrained metapath is carried out by Equation (6). We have

$$p\left(v_{i+1}\right|v_i)=\frac{1}{\left|N\right(v_i\left)\right|},(v_i,v_{i+1})\in {\mathcal{R}}^{\mathcal{C}},$$

(6)

where ${\mathcal{R}}^{\mathcal{C}}$ denotes the relation between nodes, and it is consistent with the corresponding relation-constrained metapath, i.e., the flow of the walk is constrained by the predefined metapath ${\mathcal{P}}^{\mathcal{C}}$, and $N\left(v_i\right)$ represents the neighbors of the node $v_i$.

After the relation-constrained metapath instances are constructed, we continue to filter the sequences according to the first node so that the high-order neighbors with the same node type can be directly connected with the node. Finally, we obtain homogeneous subgraphs, such as user–user or item–item subgraphs, which can capture high-order proximity between same type nodes under the different specific user behaviour. The detailed procedure is presented in Figure 2. Each homogeneous subgraph describes the potential relationship among the same type of nodes in a scene. In different scenes, the neighbors of the same node are usually different.

4.2. Node Aggregation

In order to obtain more accurate feature representation of a node, we aggregate information of its neighbors from both the intrapath and interpath view first. To be specific, the intrapath can be divided into a random walk-based intrapath and a metapath-guided intrapath. The interpath includes a random walk-based interpath and a metapath-guided interpath. Then, the final representation of a node can be trained by combining the different views organically.

4.2.1. Intrapath Aggregation

Random Walk-Based Intrapath Aggregation

The initial feature representation of the node is obtained by aggregating the neighbours in a random walk-based intrapath. Here, we use simple cumulative aggregation as shown in Equation (7). We have

$${\overrightarrow{h_v}}^t={\overrightarrow{h_v}}^{t-1}+\sum _{i\in \mathcal{N}\left(v\right)}{\overrightarrow{h_i}}^{t-1},$$

(7)

where $\overrightarrow{h_v}$ represents the vector representation of the central node, and $\mathcal{N}\left(v\right)$ represents the neighbours of the central node.

Relation Constrained Metapath-Guided Intrapath Aggregation

The subgraphs of the same node type are generated according to the high-order similar nodes in the relation-constrained metapath instances. Then, the feature vectors of neighbours are stacked to form a matrix A, which is linearly projected to three matrices Q, K, and V by use of Transformer [24]. To be specific, the feature vectors of neighbors of each node in subgraphs can merge into a feature matrix $A={[\overrightarrow{h_1},\cdots ,\overrightarrow{h_i},\cdots ,\overrightarrow{h_{l^{\prime }}}]}^T$, where $h_i\in {\mathbb{R}}^k$, k represents the feature dimension of nodes, and $l^{\prime }$ represents the number of nodes in the sequences. $A^{\prime }=\left[\overrightarrow{h_1^{\prime }},\cdots ,\overrightarrow{h_i^{\prime }},\cdots ,\overrightarrow{h_{l^{\prime }}^{\prime }}\right]$, where $\overrightarrow{h_i^{\prime }}\in {\mathbb{R}}^k$ is the vector representation of node $v_i$ after linear transformation. The converted vector representations are aggregated by Equation (8),

$$Attention(Q,K,V)=softmax\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V,$$

(8)

where $\frac{1}{\sqrt{d_k}}$ is the scaling factor. Then, we use Equation (7) to aggregate the central nodes.

4.2.2. Interpath Aggregation

After aggregation from the intrapath view, the vector representation of each node under different behaviors is obtained. All of the representations are semantic related and able to capture specific semantic information. In the next step, the different vector representations of the same node are aggregated by an attention mechanism to generate the final vector representation. Note that it is a semantic-level attention which can automatically learn the importance of different intrapaths and fuse them. Interpath aggregation includes random walk-based aggregation and relation-constrained, metapath-guided intrapath aggregation. In fact, the principle of two kinds of aggregation is the same. So we present them indiscriminately.

Firstly, we use two matrices, $W_1$ and $W_2$, which are trainable parameters acting on the same node in different scenes. Then, the attention coefficient is calculated for the same node in each scene, as shown in Equation (9),

$$e_{ij}=tanh\left(W_1\ast F\right)\ast W_2,$$

(9)

where $e_{ij}$ denotes the importance of the feature vector of the same node in the scene j to the feature vector of the scene i, and $F=\left\{\overrightarrow{f_1},\dots ,\overrightarrow{f_i},\dots ,\overrightarrow{f_n}\right\}$ is the output of previous intrapath aggregation. Next, $e_{ij}$ is normalized by the softmax function to obtain the weight ${\alpha }_{ij}$ for each scene, as shown in Equation (10),

$${\alpha }_{ij}=Softmax\left(e_{ij}\right)=\frac{exp\left({\mathit{e}}_{ij}\right)}{{\displaystyle \sum _{k\in \mathcal{R}}}exp\left({\mathit{e}}_{ik}\right)}.$$

(10)

The final representation of the interpath aggregation is computed by weighted value, as shown in Equation (11),

$$\overrightarrow{h_i}=\sigma \left(\sum _{j\in \mathcal{R}}({\alpha }_{i,j}\ast W\ast \overrightarrow{f_j})\right),$$

(11)

where $\sigma$, W represents the nonlinearity and the trainable transformation matrix, respectively.

4.3. Model Optimization

There are two kinds of walk in the proposed model. The first kind of walk is random walk, which only constrains the edge type, to generate the node sequences. All the node sequences construct the training corpus. Then, the corpus is feed into skip-gram to learn embeddings. To be specific, given a constrained relation ${\mathcal{R}}^{\mathcal{C}}$, i.e., edge type, we can get random walk-based node sequences $S_1=\left(v_{s_1},\dots ,v_{s_{l^{\prime }}}\right)$, where $l^{\prime }$ is the length of sequences, and $(v_{s_t},v_{s_{t+1}})\in {\mathcal{R}}^{\mathcal{C}}(t=1,\dots ,l^{\prime }-1)$. The context of $v_{s_t}$ can be denoted as

$$C=\left\{v_{s_k}\left|v_{s_k}\in S_1,\right|k-t\mid \le c,k\ne t\right\},$$

(12)

where c is the radius of the window size.

The second kind of walk is walking along the proposed relation-constrained metapath, which constrained both the node and edge types. Specifically, given a heterogeneous graph $G=(\mathcal{V},\mathcal{E})$ and a relation-constrained metapath ${\mathcal{P}}^{\mathcal{C}}$, a homogeneous graph can be constructed according to the Figure 2. Then, the central node and its direct neighbours can be reorganized as a node sequence, where neighbours can be considered in the context of a central node. We have $S_2=\left(v_{s_{-\frac{l^{\prime }}{2}}},\dots ,v_{s_0},\dots ,v_{\frac{l^{\prime }}{2}}\right)$, where $s_0$ denotes central node, and $(v_{s_t},v_{s_{t+1}})\in {\mathcal{R}}^{\mathcal{C}}$ and $\varphi \left(v_{s_t}\right)=\varphi \left(v_{s_{t+1}}\right)$. The context of $v_{s_0}$ can be denoted as

$$C=\left\{v_{s_k}|v_{s_k}\in S_2,|k-t\mid \le \frac{c}{2}\right\},$$

(13)

where c is the radius of window size.

Given a node $v_i$ and its context C, the objective is to minimize the negative logarithmic likelihood [5,27], as shown in Equation (14),

$$-log{P}_{\theta }\left(\left\{v_j\mid v_j\in C\right\}\mid v_i\right)=\sum _{v_j\in C}-log{P}_{\theta }\left(v_j\mid v_i\right),$$

(14)

where $\theta$ denotes all parameters. To be specific, the probability definition of $P_{\theta }\left(v_j\mid v_i\right)$ is shown in Equation (15),

$$\left.P_{\theta }\left(v_j\right|v_i\right)=\frac{exp\left(c_j^T·{\beta }_i\right)}{{\sum }_{v_k\in C}exp\left(c_k^T·{\beta }_i\right)},$$

(15)

where $c_k$ is the context embedding of node $v_k$, ${\beta }_i$ is the embedding of node $v_i$. Finally, heterogeneous negative sampling is adopted to approximate the objective function, i.e., Equation (14), for each node pair ($v_i$,$v_j$), as shown in Equation (16),

$$E=-log\sigma \left(c_j^T·{\beta }_i\right)-\sum _{n=1}^L{\mathbb{E}}_{v_k\sim P_C\left(v\right)}\left[log\sigma \left(-c_k^T·{\beta }_i\right)\right],$$

(16)

where $\sigma$ is the sigmoid function, L is the number of negative samples, $v_k$ is randomly drawn from the noise distribution $P_c\left(v\right)$ defined on node $v_j$’s corresponding to node set C.

The pseudocode of the proposed method is shown in Algorithm 1.

The time complexity in Algorithm 1 is determined by the number of nodes ($\left|\mathcal{V}\right|$), the number of edge types ($\left|\mathcal{R}\right|$), the vector dimension of node (d), and the length of metapath (l) and random walk ($l^{\prime }$), the number of negative samples per training sample (L), which is expressed as $O\left(\right|\mathcal{V}|\ast |\mathcal{R}|\ast (l+l^{\prime }+L\ast d))$. The space complexity is represented as $O\left(\right|\mathcal{V}|\ast |\mathcal{R}|\ast d))$ by the number of nodes, the vector dimension of node, and the number of edge types.    

Algorithm 1: The proposed model.

Input:  A heterogeneous graph $G=(\mathcal{V},\mathcal{E})$, a set of relation constrained metapath S, embedding dimension d, learning rate $\eta$, the length of random walk $l^{\prime }$, the number of negative samples L, window size c.

Output: Embeddings of all nodes. Initialize all the model parameters $\theta$ Generate random walk based path instances and neighbors according to Section 4.1.1; Generate relation-constrained metapath instances and neighbors according to Section 4.1.2;

4.4. Classifier for Link Prediction

Link prediction is a downstream task to evaluate the effectiveness of the learned embeddings. We consider probability-based method to evaluate it. To be specific, the sigmoid function is the final classifier in computing the performance metrics after obtaining node embeddings. The link score between node $v_i$ and $v_j$ can be computed as

$$S=sigmoid\left(\sum _{\begin{array}{c}{v}_i,v_j\\ i\ne j\end{array}}\frac{h_i^T{h}_j}{\left|\right|h_i\left|\right|\ast \left|\right|h_j\left|\right|}\right).$$

(17)

5. Simulation Experiment and Results Analysis

In this section, we aim at answering the following research questions:

• RQ1: How do different hyperparameters affect the final performance?
• RQ2: How does the proposed method perform compared with other state-of-the-art baselines on the task of link prediction?
• RQ3: How does the proposed method perform on cold-start scenarios compared to baselines?

5.1. Datasets

We use four real-world datasets to perform our experiments, including an Amazon dataset, a YouTube dataset, an Ali dataset, and a Twitter dataset. A brief description of them is given below.

Amazon dataset. The Amazon dataset [39,40] is composed of Amazon product reviews and metadata. In the experiment, only the metadata of electronic products are used. It includes two kinds of user behaviors: purchase and browsing.

YouTube dataset. The YouTube dataset [41] contains five user relationships: contacts, shared subscribers, shared subscriptions, shared friends, and shared favorite videos between users.

Ali dataset. The Ali dataset, published in Tianchi challenge (https://tianchi.aliyun.com/dataset/, accessed on 10 June 2022), contains user identity, commodity identity, and four types of user behavior, i.e.,clicking, add to favorite, add to cart and purchase. Because the dataset is too large, we randomly selected all the relevant data with a commodity class whose ID is 3424 in the experiment.

Twitter dataset. The Higgs Twitter dataset (https://snap.stanford.edu/data/higgs-twitter.html, accessed on 5 October 2022) has been built after monitoring the spreading processes on Twitter before, during, and after the announcement of the discovery of a new particle with the features of the elusive Higgs boson on 4th July 2012. It consists of four types of relations: retweeting, replying, mentioning, and social relationships among users.

The detailed statistics of datasets are summarized in

Table 2. Statistics of datasets used in experiments.

Dataset	#Nodes	#Edges	#Types of Nodes	#Types of Edges
Amazon	10,166	148,865	1	2
YouTube	2000	1,310,617	1	5
Ali	33,969	132,500	2	4
Twitter	10,000	331,899	1	4

. The structure of Amazon dataset {User Behavior, itemID1, itemID2} indicates that item 1 and item 2 are purchased or browsed by the same user at the same time. The structure of YouTube and Twitter datasets are similar to the Amazon dataset. The structure of the Ali dataset is {User behavior, userID, itemID}, which represents the relationship between user and item. The number of users and items used to train is 4099 and 17,073, respectively.

Each dataset is divided into three groups: training set, testing set, and validation set according to the ratio of 8:1:1.

5.2. Parameters Experiments (RQ1)

We introduce the parameter settings of the experiment. First, we set some fixed parameters in the experiment. The vector dimension of node in each scene is 10, and the overall dimension of the node is 200. The length of random walk is set to 10, and window size is set to 5. In order to evaluate the performance of the proposed model, ROC-AUC, PR-AUC, and F1 are used as evaluation metrics. Then, we evaluate the influence of different values of two parameters: the length of the relation-constrained metapath instance, and the number of instances per node at the start of the sequence. The experimental results of the Amazon dataset are shown in Figure 3. In Figure 3a, we observe the influence of the length of relation-constrained metapath instance by fixing the number of instances per node at the start of the sequence. To be specific, the number of instances per node is set to 60. From the figure, we have the following observation. As the length of relation-constrained metapath instance increases, the ROC-AUC, PR-AUC, and F1 values first increase, and then decrease. One possible reason for the early increase of the values of metrics is that increasing the length makes the nodes in a path instance aggregate more information from their neighborhood. Later on, the values of metrics decrease as the length of metapath instance increases, because increasing the length of metapath instance introduces some noise and reduces the semantic influence. All metrics achieve their best performance when the length of metapath instance is set to 5 on the Amazon dataset.

In order to verify the influence of the number of instances per node as the start of sequence, the length of relation constrained metapath instance is set to 5, and the experiments are carried out by changing the number of instances. As shown in Figure 3b, the values of metrics first increase, and then decrease. All the values of metrics reach the optimal value when the number of instances per node as the start of sequence is 60.

We continue to conduct experiments with the YouTube, Ali and Twitter datasets. The experimental results are shown in Figure 4, Figure 5 and Figure 6, respectively.

Finally, we draw the conclusion that our model achieves the best performances when the two variable parameters are set to (5,60), (7,50), (5,60), and (7,40) with regard to the Amazon, YouTube, Ali, and Twitter datasets, respectively.

5.3. Baselines

We regard the representative baseline approaches that fall into the following two categories. (1) Traditional models including DeepWalk [4] and node2vec [6]. (2) Heterogeneous graph-embedding models including Metapath2vec [23], GATNE [27], HetGNN [25], MAGNN [18], mSHINE [42], and GraphMSE [43].

Deepwalk [4] and node2vec [6] are two classical graph-embedding models in homogeneous graphs. Deepwalk generates a corpus on the graph by random walk, and then the corpus is fed into the skip-gram model to train the feature representation of nodes. Node2vec designs a biased random walk programming. It effectively exploits different neighbourhoods. Metapath2vec [23] is to walk regularly on the heterogeneous graph according to the designed metapath. GATNE [27] formalizes the problem of attributed multiplex embedding of heterogeneous network and supports both transductive and inductive embeddings learning for attributed multiplex heterogeneous networks. HetGNN [25] jointly considers node heterogeneous contents encoding, type-based neighbors aggregation, and heterogeneous-type combination. MAGNN [18] proposes a graph neural network based on metapath aggregation to overcome the three limitations of existing heterogeneous graph-embedding approaches in terms of features. mSHINE [42] is designed to learn multiple node representations of different metapaths simultaneously.

The hyperparameters of all baseline methods are adjusted according to the optimal parameters introduced in the papers, and the dimension of node embedding is fixed to 200.

5.4. Contrastive Experiment Results and Discussion (RQ2)

The performance of each model in link prediction task is shown in

Table 3. Performance comparison of different models on four datasets. The optimal performance and the suboptimal performance are denoted in bold and underlined fonts respectively.

Dataset	Model	ROC-AUC	PR-AUC	F1
Amazon	Deepwalk	94.20	94.03	87.38
	node2vec	94.47	94.30	87.88
	metapath2vec	94.15	94.01	87.48
	HetGNN	95.85	94.71	88.06
	GATNE	96.28	96.31	92.12
	MAGNN	87.86	87.23	85.67
	mSHINE	79.10	80.67	73.57
	Ours	97.72	97.48	93.26
YouTube	Deepwalk	71.11	70.04	65.52
	node2vec	71.21	70.32	65.36
	metapath2vec	70.98	70.02	65.34
	HetGNN	80.05	79.18	73.32
	GATNE	82.29	81.81	74.63
	MAGNN	75.16	74.69	70.13
	mSHINE	66.53	63.11	62.55
	Ours	85.23	83.85	77.13
Ali	Deepwalk	69.56	37.07	36.25
	node2vec	62.84	46.93	47.01
	metapath2vec	79.09	78.59	71.76
	HetGNN	79.95	78.24	75.37
	GATNE	82.87	85.26	76.13
	MAGNN	63.86	65.15	62.36
	mSHINE	41.07	53.04	52.43
	Ours	88.14	89.69	84.01
Twitter	Deepwalk	69.42	72.58	62.68
	node2vec	69.9	73.04	63.12
	metapath2vec	69.35	72.61	62.7
	HetGNN	72.36	75.28	69.71
	GATNE	72.4	74.4	65.89
	MAGNN	69.85	74.24	64.38
	mSHINE	67.17	70.85	63.98
	Ours	76.87	77.67	68.18

.

Compared to the GATNE model, which is almost the best baseline, we can observe that the ROC-AUC, PR-AUC, and F1 metrics are increased by 1.49%, 1.21%, and 1.23% over the Amazon dataset, and 3.57%, 2.49%, and 3.34% over the YouTube dataset, 6.35%, 5.19%, and 10.35% over the Ali dataset, and 6.17%, 4.39%, and 3.47% over the Twitter dataset, respectively.

Apparently, our model outperforms all the state-of-the-art baselines. The possible reasons are as follows. (1) Deepwalk and node2vec are classical models and adapt to the homogeneous graph. In the training process, both the node attribute and edge attribute are ignored, which leads to the loss of information. (2) The metapath2vec involves multiple edge types in the instantiation of heterogeneous graphs. Aggregation of inconsistent edge type information into a single embedding may cause noise, which degrades the performance of link prediction. (3) The GATNE model does not pay attention to the similarity between nodes and the high-order neighbour importance of the metapath in the aggregated neighbour nodes. (4) Both HetGNN and MAGNN model mainly focus on the importance of node content features, but the experimental datasets do not include node content. (5) The mSHINE model mainly aims at the incompatibility between different metapaths. However, the metapaths we selected have no such problem.

We continue to compare computational times of each method. The YouTube dataset is used as an example to report the experimental results, which are shown in Figure 7.

5.5. Cold-Start Experiment (RQ3)

The proposed model is a inductive framework, because it collects node feature information to generate node embeddings for previously unseen data, i.e., cold-start data. For example, there is little interaction data for a newly registered user, who is called cold-start user. To verify the performance of our model on a cold-start experiment, we choose GATNE model as baseline. All the interactive data with item category ID equal 3424 are randomly selected from the Ali dataset as experimental data which includes 107,730 interactive records, 4888 users and 25,682 items. The number of nodes for training, testing and validation was 21,172, 16,890 and 16,988, and the amount of interaction data for training, testing and validation was 26,561, 17,706 and 17,709, respectively. The batch-size is set to 64, the window size is set to 5, the number of instantiations per node is 30, and the length of path is 13. The optimal experimental results are achieved according to the above setting. There are a great deal of cold-start users and items. The sparse rate reaches 99.96%. Experimental results are shown in

Table 4. Effect comparison of cold-start experiment.

Metric	GATNE	Ours
ROC-AUC	85.16	88.25
PR-AUC	86.90	89.64
F1	78.27	81.72

. Comparing to GATNE, our model achieves better performance. To be specific, ROC-AUC, PR-AUC, and F1 are increased 3.62%, 3.15%, and 4.40%, respectively. It indicates that our model can better reliefs the cold start problem to a certain extent.

6. Conclusions and Future Work

In this paper, we present a novel framework of heterogeneous graph-embedding learning which consists of a novel relation constrained metapath and a representation learning module. The proposed novel relation-constrained metapath can express the characteristics between nodes more efficiently in different scenarios. The node aggregation from both intrapath and interpath views organically combines the local representation of nodes. The experimental results show that the proposed model is superior to other models. In addition, the cold-start experiment shows that our model can also relieve the cold-start problem to some extent.

In the future, there are several necessary tasks that need to be done. (1) In many real-world situations, the size of the graph is too large to be stored in memory, and we design a parallelized version for large-scale graphs. (2) The problem of metapath incompatibility is deserved to study and explore. (3) A heterogeneous contrastive learning mechanism can be integrated into our model. (4) A hyperlink in hypergraphs can connect any number of nodes [38]. An interesting idea for future research is to extend the idea of the paper to hyperlink prediction.