On Investigating Both Effectiveness and Efficiency of Embedding Methods in Task of Similarity Computation of Nodes in Graphs

Abstract

One of the important tasks in a graph is to compute the similarity between two nodes; link-based similarity measures (in short, similarity measures) are well-known and conventional techniques for this task that exploit the relations between nodes (i.e., links) in the graph. Graph embedding methods (in short, embedding methods) convert nodes in a graph into vectors in a low-dimensional space by preserving social relations among nodes in the original graph. Instead of applying a similarity measure to the graph to compute the similarity between nodes a and b, we can consider the proximity between corresponding vectors of a and b obtained by an embedding method as the similarity between a and b. Although embedding methods have been analyzed in a wide range of machine learning tasks such as link prediction and node classification, they are not investigated in terms of similarity computation of nodes. In this paper, we investigate both effectiveness and efficiency of embedding methods in the task of similarity computation of nodes by comparing them with those of similarity measures. To the best of our knowledge, this is the first work that examines the application of embedding methods in this special task. Based on the results of our extensive experiments with five well-known and publicly available datasets, we found the following observations for embedding methods: (1) with all datasets, they show less effectiveness than similarity measures except for one dataset, (2) they underperform similarity measures with all datasets in terms of efficiency except for one dataset, (3) they have more parameters than similarity measures, thereby leading to a time-consuming parameter tuning process, (4) increasing the number of dimensions does not necessarily improve their effectiveness in computing the similarity of nodes.

1. Introduction

Nowadays, graphs are becoming increasingly important since they are natural representations to encode relational structures in many domains (e.g., app’s function-call diagrams, brain-region functional activities, bio-medical drug molecules, protein interaction networks, citation networks, and social networks), where nodes represent the domain’s objects and links to their pairwise relationships [1,2,3,4,5,6,7]. Computing the similarity score between two nodes based on the graph structure is a fundamental task in a wide range of applications such as recommender systems, spam detection, graph clustering [8,9], web page ranking, citation analysis, social network analysis, k-nearest neighbor search [1,9], synonym expansion (i.e., search engine’s query rewriting and text simplification), and lexicon extraction (i.e., automatically building bilingual lexicons from text corpora) [10].

Link-based similarity measures (in short, similarity measures) such as SimRank [11] are well-known and conventional techniques to compute the similarity of nodes only based on the graph structure. Recently, SimRank and its variants have attracted a growing interest in the areas of data mining and information retrieval [1,8,9,10,12,13,14]. The philosophy of SimRank in similarity computation is that “two objects are similar if they are related to similar objects” [11]. In the literature, significant efforts have been devoted to improve the effectiveness of SimRank in similarity computation by proposing different variants such as SimRank++ [15], PSimRank [16], JacSim [1], JPRank [17], RoleSim [18], MatchSim [19], C-Rank [20], SimRank* [9], SimRank# [12], and PRank [21]. Among the aforementioned SimRank variants, JacSim, SimRank*, and JPRank are the state-of-the-art ones; JacSim solves the pairwise normalization problem [1,16,20], SimRank* solves the level-wise computation problem [9], and JPRank remedies both the pairwise normalization and in-links consideration problems [17,21] in the original SimRank. In addition, other variants such as GSimRank [14] and HowSim [8] have been proposed to compute the similarity of nodes in heterogeneous graphs where nodes have different types and links represent different types of relationships among nodes (However, in this paper, we only focus on the homogeneous graphs where nodes and links have a single type such as a citation graph where nodes represent papers and links to the citation relationship among them).

Graph embedding methods (in short, embedding methods), also known as feature representation learning methods, exploit the graph structure to represent each node in the graph as a low-dimensional vector in which neighborhood similarity, semantic information, and community structure among nodes in the graph are captured [22,23,24,25]. The obtained vector representations can be utilized by a wide range of tasks such as link prediction [22,24,26,27,28], node classification [22,23,25,26,27,28,29,30], recommendations [27], word analogy detection [25], and document classification [25]. Embedding methods, which are effective for extracting features from graph structured data, have broadly attracted significant attention in the literature and different embedding methods such as DeepWalk [23], Line [25], node2vec [26], graphGAN [27], NetMF [29], ATP [24], BoostNE [30], DWNS [22], and NERD [28] have been proposed for homogeneous graphs. In addition, some embedding methods have been proposed that target the heterogeneous graphs such as HeGAN [31], RTN [32], and HAN [33]. Some methods such as SIDE [34], CSNE [35], and nSNE [36] perform feature representation learning in signed graphs where there are two types of links (i.e., positive and negative) (In this paper, we focus only on the unsigned homogeneous graphs and the related embedding methods).

All of the aforementioned embedding methods encode the social relations among nodes into vectors in a continuous low-dimensional space where each dimension in this space can be interpreted as a latent feature [22,23,24,26]. In order to compute the similarity score of a pair of nodes in a given graph, we can apply Cosine [37] to their corresponding vector representations, similar to the strategy observed in the word analogy detection [25,38]. In other words, a link-based similarity measure computes the similarity between nodes by directly exploiting the graph structure, while an embedding method converts a node in the graph into a vector representation composed of a set of latent features by exploiting the graph structure, then these latent features preserving the graph structure properties can be used to compute the similarity between nodes.

Although a lot of efforts have been devoted to propose powerful embedding methods in the literature by analyzing them in a wide range of machine learning tasks, they are not comprehensively analyzed in terms of similarity computation of nodes in graphs. In this paper, we investigate and analyze both the effectiveness (i.e., accuracy) and efficiency (i.e., execution time) of embedding methods (i.e., DeepWalk, Line, node2vec, graphGAN, NetMF, ATP, BoostNE, DWNS, and NERD) in similarity computation of nodes and compare them with those of similarity measures (i.e., SimRank, JacSim, JPRank, and SimRank*). To the best of our knowledge, this is the first work that examines the application of embedding methods in computing the similarity of nodes. We conduct extensive experiments with five well-known and publicly available datasets as BlogCatalog [23,27,29], Cora [39,40], DBLP [1,17,41], TREC [1,17], and Wikipedia [26,27,29], which are widely used in the literature. Our experimental results demonstrate that similarity measures are better than embedding methods to compute the similarity of nodes; we found the following observations for embedding methods: (1) with all datasets, they show less effectiveness than similarity measures except for the BlogCatalog dataset; however, they show less efficiency than similarity measures with this dataset, (2) they underperform similarity measures with all datasets in terms of efficiency except for the Wikipedia dataset; however, with this dataset, they show less effectiveness than similarity measures, (3) they have more parameters than similarity measures, thereby leading to a difficult and time-consuming parameter tuning process, and (4) increasing the number of dimensions (i.e., latent features) does not necessarily improve their effectiveness in computing the similarity of nodes. In addition, we observe that, among embedding methods, DeepWalk and its variants (i.e., Line and NetMF) show better effectiveness.

The contributions of this paper are summarized as follows:

• Although embedding methods have been analyzed in a wide range of machine learning tasks, they are not investigated in terms of similarity computation of nodes. We investigate and analyze both the effectiveness and efficiency of embedding methods in the task of similarity computation of nodes in graphs.
• We compare the effectiveness as well as efficiency of embedding methods with those of link-based similarity measures as the conventional technique to compute the similarity of nodes.
• We conduct extensive experiments with five widely used datasets by employing nine different embedding methods and four different similarity measures, which all are state-of-the-art ones in the literature.

The rest of this paper is organized as follows. We discuss link-based similarity measures and graph embedding methods in Section 2 and Section 3, respectively. In Section 4, we present and discuss the results of our extensive experiments. In Section 5, we conclude our paper.

2. Link-Based Similarity Measures

In this section, we briefly explain SimRank [11], a well-known link-based similarity measure (in short, similarity measure), and its state-of-the-art variants as JacSim [1], JPRank [17], and SimRank* [9]; their corresponding mathematical formulations are represented in Appendix A.1, in detail.

SimRank: it computes the similarity between two nodes in a graph based on a philosophy that “two objects are similar if they are related to similar objects” [11]. For a node-pair $(a,b)$ in a graph, let $I_a$ and $I_b$ be two sets of nodes directly pointing to nodes a and b, respectively. In SimRank, the similarity score of $(a,b)$ is iteratively computed as the average of similarity scores of all possible node-pairs$(i,j)$ where node i belongs to $I_a$ and node j belongs to $I_b$; this computation manner is called a pairwise normalization paradigm [1]. Consider a sample graph in Figure 1; SimRank considers nodes e and f similar since they are directly pointed to by common node b (each node is highly similar to itself) and nodes i and j are regarded as similar since they are indirectly pointed to by common nodes c and d.

JacSim: it tries to solve the pairwise normalization problem, by employing both Jaccard and pairwise normalization paradigm in similarity computation. This problem is a counter-intuitive property of SimRank where the SimRank score of a pair of nodes commonly pointed to by a large number of nodes tends to be lower than that of another pair of nodes commonly pointed to by a small number of nodes [1,16,20]. As an example, consider node-pairs $(i,h)$ and $(i,j)$ in the graph of Figure 1 where the similarity scores of some node-pairs are shown as well (We note that all the scores are computed by employing the matrix form of SimRank, JacSim, and JPRank); nodes i and h are pointed to by a single common node b, while nodes i and j are pointed to by two common nodes c and d. However, the SimRank score of $(i,h)$ (i.e., 0.0106) is bigger than that of $(i,j)$ (i.e., 0.0071) due to the pairwise normalization problem, while, as shown in the figure, the JacSim score of $(i,h)$ (i.e., 0.0048) is smaller than that of $(i,j)$ (i.e., 0.0076).

SimRank*: it is a variant of SimRank trying to remedy the level-wise computation problem in SimRank; this problem happens since SimRank regards two nodes similar if some paths only with equal length exist from a common node to both of them. As an example, consider the node-pair $(e,i)$ in Figure 1; there are no paths with the equal length from any common nodes such b to them. There is a path with length two from b to i, while there is a path with length one from b to e. Therefore, the SimRank score of $(e,i)$ becomes zero as shown in the figure due to the level-wise computation problem; however, their SimRank* score is not zero (i.e., 0.0206).

JPRank: it is another variant of SimRank that solves both the pairwise normalization and in-links consideration problems. The latter problem arises since SimRank considers only in-links to compute the similarity score. As an example, in Figure 1, the SimRank score of node-pair $(b,c)$ is zero since $I_b=I_c=\varnothing$. However, both b and c are pointing to a common node f, which means b and c are somehow similar; as shown in the figure, the JPRank score of $(b,c)$ is not zero (i.e., 0.0028).

It is worth noting that, although our sample graph in Figure 1 is directed, all the above similarity measures can be applied to undirected graphs as well. However, the in-links consideration problem is not applicable to an undirected graph and JPRank behaves exactly the same as JacSim in similarity computation [17].

3. Graph Embedding Methods

In this section, we briefly describe the concept of graph embedding and explain some of the state-of-the-art graph embedding methods (in short, embedding methods) in the literature; their corresponding objective functions are represented in Appendix A.2, in detail.

For a given graph $G=(V,E)$, graph embedding aims to learn a function $f:V\to R^d$ that maps each node v in the graph into a vector in the d-dimensional space where $d\ll \left|V\right|$ [23,24,25,29]. The embedding methods exploit the graph structure to represent nodes as a low-dimensional vectors that encode the neighborhood similarity, semantic information, and community structure among nodes in the graph [22,23,24,25]. In the low-dimensional space, each dimension of the vectors can be interpreted as a latent feature [22,23,24,26]. Figure 2a illustrates the Zachary’s Karate graph [42] where the clusters found by modularity maximization are shown in different colors. We applied DeepWalk on this graph to obtain its two-dimensional representation, which is shown in Figure 2b; as an example, nodes 1 and 2 in the graph are represented as <−1.18, 1.15> and <−0.30, 1.10> vectors, respectively. As observed in the figure, there is a commonality between clusters in the original graph and its representation since they encode the social relations and community structure in the graph. Therefore, in order to compute the similarity score of node-pair $(a,b)$ in a given graph, we can calculate the proximity of the corresponding vector representations of a and b, similar to the strategy observed in the word analogy detection [25,38].

2

DeepWalk [23]: inspired by the remarkable achievements in the representation learning for natural language processing such as Skip-gram [38], it exploits a stream of short random walks to extract information from a graph where these short random walks can be regarded as a neighborhood for a target node $v_i$ consisting of n nodes before and after $v_i$ (i.e., window size $W=2n$). DeepWalk tries to learn a model to maximize the probability of any node appearing in the $v_i$’s neighborhood without the knowledge of its offset from $v_i$.

Line [25]: it considers the first-order proximity (i.e., direct neighbors of a node) and the second-order proximity to capture the nodes’ neighborhoods; the second-order proximity follows the sociology and linguistics theories where two nodes sharing similar neighbors tend to be similar. These proximities are preserved by utilizing two distinct objective functions where two different models for them are trained separately and their results are concatenated as the final result.

node2vec [26]: by following the homophily and structural equivalence hypotheses, it considers the community of a node and its structural roles in the graph, respectively, to capture the node neighborhood. node2vec utilizes a biased random walk controlled by two parameters p (return parameter) and q (in-out parameter). Suppose a random walk by using link $(u,v)$ in a graph; to decide the next walk from node v, parameter p controls the probability of revisiting previous node in the walk (i.e., u) and parameter q controls the probability of visiting a node close to v or visiting a node that is far from v.

graphGAN [27]: it considers two models, generator $\mathbb{G}$ and discriminator $\mathbb{D}$, involving in a minimax game as follows. For a target node v, the generator tries to generate relevant nodes to the v’s neighborhood (i.e., found by BFS search rooted in v) and produces fabricated samples to deceive the discriminator, while the discriminator tries to detect whether a node actually belongs to v’s neighborhood or it is fabricated by the generator. In graphGAN, the objective is to train two models as a two-play minimax game.

NetMF [29]: it shows that graph embedding methods utilizing the Skip-gram model (e.g., DeepWalk, Line, and node2vec) with negative sampling perform implicit matrix factorization with a closed form. NetMF draws a theoretical connection between DeepWalk’s implicit matrix and graph Laplacians leading to constructing a low-rank connectivity matrix for DeepWalk, which is explicitly factorized by the singular value decomposition (SVD) technique [43] to obtain the representation vectors.

ATP [24]: it is a matrix-factorization-based embedding method that tries to preserve the asymmetric transitivity property in community question answering (CQA) graphs (i.e., if question $q_1$ is easier than $q_2$ and $q_2$ is easier than $q_3$, then $q_1$ is easier than $q_3$). ATP constructs graph $G^{\prime }$ by removing cycle links from G, and incorporates both the graph reachability (i.e., transitive closure of $G^{\prime }$ [24]) and hierarchy (i.e., the rank of nodes in $G^{\prime }$) into a single matrix, which is factorized by a non-negative matrix factorization (NMF) technique [44] to obtain the representation vectors; each node has two corresponding vectors since its role in the graph is regarded as both a source and a target.

BoostNE [30]: it is a matrix-factorization-based embedding method that does not regard the low-rank assumption on the DeepWalk’s connectivity matrix M; applying a single NMF on M may lead to obtain such representations, which are insufficient to encode the connectivity patterns among nodes. Inspired by ensemble learning methods [45], BoostNE performs multiple levels of NMF on M to construct the representation vectors.

DWNS [22]: it applies the adversarial training method [46] to DeepWalk in order to improve the robustness and generalization ability of the learning process; it forces the learned classifier to be robust to adversarial examples (i.e., fabricated samples) generated from real ones through small perturbations. The training process is a two-player game where the adversarial samples are generated to maximize the model loss while the embedding vectors are optimized against them by utilizing stochastic gradient descent (SGD) [47].

NERD [28]: it mainly considers two different roles, a source and a target, for any nodes in the graph and maintains separate embedding spaces for the two distinct roles; based on this consideration, it exploits two distinct neighborhoods for each node and tries to maximize the likelihood of preserving both neighborhoods in their corresponding embedding spaces at the learning process.

4. Experimental Evaluation

In this section, we extensively analyze both the effectiveness (i.e., accuracy) and efficiency (i.e., execution time) of embedding methods (i.e., DeepWalk, Line, node2vec, graphGAN, NetMF, ATP, BoostNE, DWNS, and NERD) in the task of similarity computation of nodes in graphs with those of similarity measures (i.e., SimRank, JacSim, JPRank, and SimRank*) as the conventional technique for this task. Section 4.1 describes our experimental setup; Section 4.2 presents and analyzes the results.

4.1. Experimental Setup

All our experiments are performed on an Intel machine equipped with six 3.60 GHz i5-8600 CPUs, 64 GB RAM, and a 64-bit Fedora Core 31 operating system. All required codes are implemented with Python. We employ five well-known and publicly available datasets for our evaluation as follows.

Table 1. Some statistics about our datasets.

	$\left|\mathit{V}\right|$	$\left|\mathit{E}\right|$	#Labels	Graph Type	Label Type
BlogCatalog	10,312	333,982	39	undirected	full
Cora	23,166	91,500	70	directed	full
DBLP	21,177	248,131	11	directed	partial
TREC	43,202	347,702	11	directed	partial
Wikipedia	4777	184,812	40	undirected	full

shows some statistics of our datasets:

• BlogCatalog [23,27,29] is a graph representing social relationships among bloggers. The node labels denote blogger interests inferred through the metadata provided by the bloggers. This graph is fully tagged by 39 different labels.
• Cora [39,40] is a citation graph of academic papers in the area of computer science. The node labels denote the paper’s topic (e.g., Networking-Protocols). This graph is fully tagged by 70 different labels.
• DBLP [1,17,41] is a citation graph of academic papers in the areas of data mining and databases published in 2006 and earlier. The node labels denote the paper’s topic. This graph is partially tagged (i.e., 200 labeled nodes) by 11 different labels.
• TREC [1,17] is a hyperlink graph based on TREC 2003) (http://trec.nist.gov/data.html) where nodes represent web pages and links to the hyperlinks between web pages. The node labels denote the web page’s topic. This graph is partially tagged (i.e., 127 labeled nodes) by 11 different labels.
• Wikipedia [26,27,29] is a co-occurrence graph of words appearing in the first million bytes of the English Wikipedia dump. The labels represent the inferred Part-of-Speech (POS) tags of words. This graph is fully tagged by 40 different labels.

In the case of undirected graphs, we create two links in both directions. In order to evaluate the effectiveness (i.e., accuracy), we utilize MAP (mean average precision), precision, recall, F-score [37], and PRES [48] as evaluation metrics. In each dataset, we consider the labels as ground truth; for each label l, we use every single node with label l as a query node for a similarity based searching, and find those nodes that are considered similar to the query as a result set. If a node in the result set is labeled with l, it is regarded as relevant to the query, otherwise irrelevant. Then, we compute precision, recall, F-score, average precision (AP), and PRES for that query as follows:

$$precision=\frac{|Res\cap Rel|}{\left|Res\right|}$$

(1)

$$recall=\frac{|Res\cap Rel|}{\left|Rel\right|}$$

(2)

$$F-score=\frac{2\times recall\times precision}{recall+precision}$$

(3)

where $Res$ indicates the query result set, $Rel$ indicates the set of relevant nodes to the query (i.e., the set of all nodes labeled with l). $\left|Rel\right|$ and $\left|Res\right|$ indicate the sizes of $Rel$ and $Res$, respectively. In the AP measure, a precision value is computed in each position (rank) in the query result set:

$$AP=\frac{{\sum }_{k=1}^{\left|Res\right|}P@k\times Rel\left(k\right)}{\left|Rel\right|}$$

(4)

where $P@k$ indicates the precision at position k, $Rel\left(k\right)$ is set as 1 if the node in position k is regarded as relevant. Otherwise, it is set as 0.

PRES considers the rank of retrieved relevant nodes in a result set and is computed as follows:

$$PRES=1-\frac{\frac{\mathrm{\Sigma }{r}_i}{\left|Rel\right|}-\frac{\left|Rel\right|+1}{\left|2\right|}}{\left|Res\right|}$$

(5)

where $r_i$ is the rank of the $i\mathrm{th}$ relevant node in the result set; for each of those m (i.e., $m\le \left|Rel\right|$) number of nodes that are relevant to the query but not retrieved in the result set, a rank is assigned by starting from the value of $\left(\right|Res|+|Rel|-x+1)$.

After computing the AP, precision, recall, F-score, and PRES for all the query nodes with label l, we take their average values to get MAP, precision, recall, F-score, and PRES values for label l. Then, we compute the average values of MAP, precision, recall, F-score, and PRES over all labels in the dataset. The aforementioned process is separately computed at top t (t = 5, 10, 20, 30) results; finally, the average accuracy over all values of t (e.g., we calculate the MAP value by taking the average of four MAP values at top 5, 10, 20, and 30 results) is regarded as the final accuracy for the dataset.

We implemented the matrix forms of SimRank, JacSim, JPRank, and SimRank* by applying the default parameter settings suggested by their original work with all datasets. We set the impact factor C as 0.8 for all four measures. For JacSim, the importance factor $\alpha$ is set as 0.4 by following [1]. For JPRank, both importance factors ${\alpha }_1$ and ${\alpha }_2$ are set as 0.4 by following [17]; however, to indicate the value of weighting parameter $\beta$, we conducted a very simple and fast experiment by following [17] to find out between in-links and out-links which one is more beneficial to similarity computation as follows. With each dataset, we computed SimRank based on in-links and out-links only on four iterations and then compared the accuracy of these two computations; $\beta$ is set as 0.9 if the similarity computation based on in-links shows better accuracy (i.e., with the DBLP dataset); it is set as 0.1, otherwise (i.e., with the TREC and Cora datasets (As explained in Section 2, JPRank behaves the same as JacSim with undirected graphs; we do not apply it to these graphs)).

In the case of embedding methods, we utilized the implementations with the default parameter settings provided by their original work (i.e., ATP (https://github.com/zhenv5/atp#atp-directed-graph-embedding-with-asymmetric-transitivity-preservation), BoostNE (https://github.com/benedekrozemberczki/BoostedFactorization), DeepWalk (https://github.com/phanein/deepwalk), DWNS (https://github.com/wonniu/AdvT4NE_WWW2019), graphGAN (https://github.com/hwwang55/GraphGAN), Line (https://github.com/tangjianpku/LINE), NERD (https://git.l3s.uni-hannover.de/khosla/nerd), NetMF (https://github.com/xptree/NetMF), and node2vec (https://github.com/aditya-grover/node2vec)). For ATP, we applied the H-voting technique (https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchies) (it is not provided by the original implementation of ATP) to the datasets to find their cycle links. We could not apply graphGAN to Cora and TREC datasets since graphGAN requires lots of memory, and it does not run with these datasets on a machine with 64 GB memory. In order to compute the similarity between low-dimensional vectors, we implemented Cosine based on a matrix/vector multiplication technique, which is significantly faster than its conventional implementation. We made publicly available our implementations of the four similarity measures and Cosine along with the five aforementioned datasets here (https://github.com/mrhhyu/EMB_vs_LB).

4.2. Results and Analyses

In Section 4.2.1, for each dataset, we find the best iterations on which similarity measures show their highest accuracies. In Section 4.2.2, for each dataset, we find the best values of d (i.e., number of dimensions) for which the embedding methods show their highest accuracies in similarity computation of nodes. In Section 4.2.3 and Section 4.2.5, we present an experimental analysis on the effectiveness and efficiency of embedding methods (i.e., based on their best values of d) in comparison with similarity measures (i.e., based on their best iterations), respectively. Section 4.2.4 analyzes the impact of the value of d on the accuracy of embedding methods.

4.2.1. Link-Based Similarity Measures: Best Iterations

We apply the similarity measures to our five datasets on eight iterations; then, for each similarity measure with a dataset, we find out the best iteration on which the similarity measure shows its highest accuracy. Figure 3 illustrates the results with our five datasets; as an example, SimRank shows its highest accuracy on iterations 2 and 3 with the BlogCatalog and DBLP datasets, respectively. As already noted in Section 2, JPRank shows exactly the same results as JacSim does with undirected graphs; therefore, we do not apply JPRank to the BlogCatalog and Wikipedia datasets (i.e., 5 ∗ 4 − 2 = 18 experimental cases are conducted). In addition, in the figure, we do not represent the precision metric since the range of its values is higher than MAP, recall, PRES, and F-score; it makes the other four metrics plotted out very close together in the figure, thereby decreasing the readability of figures.

As shown in Figure 3, for all similarity measures, the best iteration is observed before the eighth one in all datasets.

Table 2. Best iterations of all similarity measures.

	JacSim	JPRank	SimRank*	SimRank
BlogCatalog	1	1	2	2
Cora	5	4	5	5
DBLP	3	3	6	3
TREC	2	2	2	1
Wikipedia	5	5	2	2

summarizes the best iterations for all similarity measures with our datasets. Note that, hereafter, when we compare the effectiveness of a similarity measure with those of embedding methods for a dataset, we consider the effectiveness of the similarity measure on its best iteration with that dataset; as an example, in the case of SimRank with the BlogCatalog dataset, we consider its effectiveness on the second iteration according to Table 2.

4.2.2. Graph Embedding Methods: Best Values of d

Now, we apply ATP, BoostNE, DeepWalk, DWNS, graphGAN, Line, NERD, NetMF, and node2vec to our five datasets to obtain the low-dimensional representation vectors of the nodes. Then, to compute the similarity between two nodes in a dataset, we apply Cosine to their corresponding vectors. In order to carefully analyze the impact of the number of dimensions d in similarity computation of nodes, we set d to different values as 64, 128, 256, and 512. For each possible combination of methods, datasets, and d values (e.g., ATP with the BlogCatalog dataset when $d=64$), we perform the experiment five times and select the best accuracy obtained among these five different executions as the final accuracy for that combination. More specifically, we conducted 900 ($=5\times 9\times 5\times 4$) different experiments. Finally, similar to the strategy taken in Section 4.2.1, in the case of each embedding method with a dataset, we find out the best value of d for which the embedding method shows its highest accuracy in similarity computation of nodes.

Figure 4 and Figure 5 illustrate the accuracy of embedding methods with different values of d. The former figure shows the results with the BlogCatalog, Cora, and DBLP datasets, while the latter one shows the results with the TREC and Wikipedia datasets; as an example, BoostNE shows its highest accuracy when d is set as 256 and 128 with the BlogCatalog and TREC datasets, respectively. In these figures, we do not represent the precision metric due to the same reason as in Figure 3; the values of this metric with BlogCatalog, Cora, DBLP, TREC, and Wikipedia datasets for different values of d are represented in

Table A1. Precision values of all embedding methods with the BlogCatalog dataset and different values of d.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec
$d=64$	0.083	0.039	0.161	0.070	0.047	0.105	0.037	0.155	0.117
$d=128$	0.083	0.041	0.161	0.084	0.044	0.101	0.049	0.167	0.116
$d=256$	0.084	0.045	0.156	0.096	0.039	0.101	0.048	0.17	0.115
$d=512$	0.085	0.041	0.152	0.101	0.041	0.100	0.048	0.167	0.115

,

Table A2. Precision values of all embedding methods with the Cora dataset and different values of d.

	ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec
$d=64$	0.259	0.081	0.401	0.355	0.391	0.085	0.070	0.231
$d=128$	0.302	0.105	0.401	0.367	0.391	0.283	0.075	0.231
$d=256$	0.313	0.121	0.399	0.376	0.391	0.28	0.075	0.228
$d=512$	0.325	0.129	0.398	0.376	0.391	0.28	0.075	0.224

,

Table A3. Precision values of all embedding methods with the DBLP dataset and different values of d.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec
$d=64$	0.046	0.024	0.107	0.074	0.036	0.050	0.013	0.019	0.036
$d=128$	0.053	0.033	0.114	0.064	0.031	0.053	0.078	0.022	0.040
$d=256$	0.057	0.028	0.107	0.081	0.029	0.052	0.072	0.021	0.039
$d=512$	0.058	0.041	0.097	0.064	0.030	0.052	0.079	0.023	0.035

,

Table A4. Precision values of all embedding methods with the TREC dataset and different values of d.

	ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec
$d=64$	0.018	0.004	0.033	0.026	0.021	0.001	0.032	0.015
$d=128$	0.017	0.008	0.033	0.015	0.021	0.005	0.033	0.021
$d=256$	0.017	0.005	0.031	0.019	0.022	0.005	0.032	0.018
$d=512$	0.018	0.001	0.032	0.025	0.022	0.005	0.034	0.012

and

Table A5. Precision values of all embedding methods with the Wikipedia dataset and different values of d.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec
$d=64$	0.041	0.04	0.056	0.041	0.035	0.06	0.045	0.056	0.052
$d=128$	0.041	0.043	0.052	0.044	0.036	0.058	0.049	0.053	0.051
$d=256$	0.041	0.043	0.05	0.046	0.034	0.056	0.048	0.05	0.051
$d=512$	0.041	0.045	0.051	0.048	0.034	0.057	0.049	0.05	0.05

in Appendix A, respectively. As already noted in Section 4.1, we cannot apply graphGAN to the Cora and TREC datasets due to their large sizes.

Table 3. Best values of d for all embedding methods.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec
BlogCatalog	512	256	128	512	64	512	128	256	128
Cora	512	512	512	512	-	512	128	128	64
DBLP	512	512	128	256	64	128	128	512	128
TREC	512	128	64	64	-	256	256	512	128
Wikipedia	512	128	64	256	64	64	512	64	64

summarizes the best value of d for all embedding methods with our datasets. Note that, hereafter, when we compare the effectiveness of embedding methods with those of similarity measures for a dataset, we consider the effectiveness of embedding methods on their best values of d with that dataset; as an example, in the case of DeepWalk with the BlogCatalog dataset, we consider its effectiveness based on $d=128$ according to Table 3.

4.2.3. Effectiveness Evaluation

In this section, we analyze the effectiveness (i.e., accuracy) of embedding methods in computing the similarity of nodes and compare it with those of similarity measures with each dataset as follows. As explained in Section 4.2.1 and Section 4.2.2, to compare the effectiveness of similarity measures with embedding methods for each dataset in this section, we consider their accuracies on their best iterations and best values of d represented in Table 2 and Table 3, receptively.

• BlogCatalog Dataset

Figure 6 illustrates the accuracy of all embedding methods and similarity measures with the BlogCatalog dataset. In this figure, we do not represent the precision measure due to the same reason as in Figure 3; instead, we show the precision values for all methods in

Table 4. Precision values with the BlogCatalog dataset.

ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec	JacSim	SimRank*	SimRank
0.085	0.045	0.161	0.101	0.047	0.100	0.049	0.170	0.116	0.090	0.125	0.069

. In addition, for those embedding methods and similarity measures that show comparable accuracies, we write down the values of their corresponding MAP, PRES, recall, and F-score in the figure to have better comparison.

As observed in the figure, with the BlogCatalog dataset, NetMF shows the highest accuracy among all the embedding methods in terms of MAP, precision, recall, PRES, and F-score; however, its accuracy is close to that of DeepWalk, while BoostNE, graphGAN, and NERD show the worst accuracies. SimRank* shows better accuracy than other similarity measures, while SimRank shows the worst accuracy in terms of MAP, precision, recall, PRES, and F-score. Now, by comparing the accuracy of NetMF with that of SimRank*, it is observed that NetMF outperforms SimRank* by 73.44%, 35.68%, 42.89%, 50.00%, and 47.10% in terms of MAP, precision, recall, PRES, and F-score, respectively;

Table 5. Accuracy improvements (%) by NetMF over other methods with the BlogCatalog dataset.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	node2vec	JacSim	SimRank*	SimRank
MAP	209.77	1010.00	14.83	154.20	1068.42	163.24	838.03	121.26	179.83	73.44	365.73
precision	100.72	280.94	5.22	68.06	286.49	69.75	246.76	46.55	89.36	35.68	147.22
recall	162.84	456.31	8.73	99.65	519.46	114.61	293.81	88.18	125.15	42.89	245.18
PRES	168.22	617.50	9.68	112.59	682.73	128.99	431.48	101.17	145.30	50.00	286.10
F-score	147.01	418.26	9.88	106.51	466.07	113.95	344.39	88.32	135.98	47.10	222.37

shows the percentage of improvements in accuracy obtained by NetMF over all other methods with the BlogCatalog dataset.

• Cora Dataset

Figure 7 illustrates the accuracy of all embedding methods and similarity measures with the Cora dataset and

Table 6. Precision values with the Cora dataset.

ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec	JacSim	JPRank	SimRank*	SimRank
0.325	0.129	0.398	0.376	0.391	0.283	0.075	0.231	0.198	0.414	0.360	0.191

shows the precision values for all methods.

As observed in the figure, with the Cora dataset, although DeepWalk and Line show the best accuracy among all embedding methods, their accuracies are not tangible; Line outperforms DeepWalk in terms of recall, PRES, and F-score, while DeepWalk outperforms Line in terms of MAP and precision. NetMF and BoostNE show the worst accuracy among embedding methods in terms of all metrics. In the case of similarity measures, JPRank shows the best accuracy and SimRank again shows the worst one in terms of MAP, precision, recall, PRES, and F-score. Now, by comparing the accuracy of Line with that of JPRank, it is observed that JPRank slightly outperforms Line by 4.05%, 5.73%, 2.28%, 2.98%, and 3.05% in terms of MAP, precision, recall, PRES, and F-score, respectively;

Table 7. Accuracy improvements (%) by JPRank over other methods with the Cora dataset.

	ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec	JacSim	SimRank*	SimRank
MAP	36.37	416.85	3.69	16.47	4.05	79.21	622.32	157.58	112.41	16.82	128.88
precision	27.36	221.00	3.87	9.98	5.73	46.49	452.53	79.41	109.17	14.84	116.91
recall	31.22	298.02	3.23	12.33	2.28	57.35	566.12	112.11	106.53	16.90	111.13
PRES	32.92	343.59	3.41	13.99	2.98	66.69	579.35	130.91	107.46	15.85	116.63
F-score	31.99	286.92	3.50	12.64	3.05	56.18	551.31	105.21	110.18	17.76	115.65

shows the percentage of improvements in accuracy obtained by JPRank over all other methods with the Cora dataset.

• DBLP Dataset

Figure 8 illustrates the accuracy of all embedding methods and similarity measures with the DBLP dataset and

Table 8. Precision values with the DBLP dataset.

ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec	JacSim	JPRank	SimRank*	SimRank
0.058	0.041	0.114	0.081	0.036	0.053	0.078	0.022	0.040	0.168	0.172	0.094	0.094

shows the precision values for all methods.

As observed in the figure, DeepWalk shows the best accuracy among all embedding methods, while NetMF shows the worst one in terms of MAP, precision, recall, PRES, and F-score. Among similarity measures, JPRank shows the best accuracy and it is close to that of JacSim, while SimRank shows the worst accuracy in terms of all metrics. Now, by comparing the accuracy of DeepWalk with that of JPRank, it is observed that JPRank outperforms DeepWalk by 64.89%, 50.34%, 51.70%, 52.71%, and 51.11% in terms of MAP, precision, recall, PRES, and F-score, respectively;

Table 9. Accuracy improvements (%) by JPRank over other methods with the DBLP dataset.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec	JacSim	SimRank*	SimRank
MAP	306.56	441.78	64.89	168.03	603.43	363.51	150.31	1042.26	373.62	1.77	135.37	168.23
precision	195.25	314.09	50.34	111.59	382.84	226.79	119.44	665.05	329.97	2.44	82.36	82.17
recall	183.36	300.94	51.70	109.85	382.53	210.52	119.18	668.47	375.16	2.46	60.41	59.66
PRES	213.05	343.68	52.71	120.69	438.64	262.21	125.86	705.20	375.48	2.05	90.03	94.22
F-score	188.63	307.11	51.11	110.32	382.18	216.87	118.79	663.16	353.98	2.47	69.03	67.67

shows the percentage of improvements in accuracy obtained by JPRank over all other methods with the DBLP dataset.

• TREC Dataset

Figure 9 illustrates the accuracy of all embedding methods and similarity measures with the TREC dataset and

Table 10. Precision values with the TREC dataset.

ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec	JacSim	JPRank	SimRank*	SimRank
0.018	0.008	0.033	0.026	0.022	0.005	0.034	0.021	0.019	0.031	0.035	0.015

shows the precision values for all methods.

As observed in the figure, DeepWalk and NetMF show the best accuracy among all embedding methods in terms of MAP, precision, recall, PRES, and F-score. However, their accuracies are not tangible, where NetMF outperforms DeepWalk in terms of precision, PRES, and F-score, while DeepWalk shows better accuracy in terms of MAP and recall. NERD shows the worst accuracy among all embedding methods. SimRank* shows the best accuracy among all similarity measures, while SimRank shows the worst one in terms of all metrics. Now, by comparing the accuracy of NetMF with that of SimRank*, it is observed that they show very close accuracy; SimRank* outperforms NetMF in terms of precision, recall, and F-score, while NetMF outperforms SimRank* in terms of MAP and PRES.

Table 11. Accuracy improvements (%) by SimRank* over other methods with the TREC dataset.

	ATP	BoostNE	DeepWalk	DWNS	Line	NERD	NetMF	node2vec	JacSim	JPRank	SimRank
MAP	156.37	304.33	−9.82	45.81	15.95	617.85	−7.38	69.67	79.46	−23.56	211.07
precision	94.61	315.85	3.46	35.58	58.54	548.69	2.97	64.33	78.74	12.43	127.45
recall	97.67	346.31	5.82	37.06	81.61	780.16	6.03	71.36	64.31	31.32	91.21
PRES	110.68	343.59	−0.37	36.54	49.24	662.63	−0.73	64.85	77.88	6.68	132.30
F-score	94.42	331.43	3.74	34.57	67.83	651.72	3.52	66.84	71.41	20.90	107.64

shows the percentage of improvements in accuracy obtained by SimRank* over all other methods with the TREC dataset.

• Wikipedia Dataset

Figure 10 illustrates the accuracy of all embedding methods and similarity measures with the Wikipedia dataset, and

Table 12. Precision values with the Wikipedia dataset.

ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec	JacSim	SimRank*	SimRank
0.041	0.043	0.056	0.046	0.035	0.060	0.049	0.056	0.052	0.075	0.039	0.040

shows the precision values for all methods. As observed in the figure, DeepWalk shows the highest accuracy among embedding methods in terms of MAP, precision, recall, PRES, and F-score; however, its accuracy is close to that of DWNS, while graphGAN shows the worst accuracy. Among similarity measures, JacSim shows the best accuracy, while SimRank shows the worst one in terms of all metrics. Now, by comparing the accuracy of DeepWalk with that of JacSim, it is observed that JacSim outperforms DeepWalk by 140.20%, 34.91%, 74.16%, 83.45%, and 66.55% in terms of MAP, precision, recall, PRES, and F-score;

Table 13. Accuracy improvements (%) by JacSim over other methods with the Wikipedia dataset.

	ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec	SimRank*	SimRank
MAP	458.30	502.50	140.20	168.77	1663.41	179.69	557.27	161.48	330.36	347.68	1125.42
precision	80.82	72.57	34.91	61.70	112.21	24.10	54.15	33.68	43.00	90.86	85.06
recall	405.74	204.50	74.16	54.05	593.77	98.22	354.20	105.76	242.50	82.38	702.54
PRES	372.19	266.90	83.45	91.50	913.74	114.11	375.33	105.08	247.24	181.82	762.50
F-score	252.98	212.50	66.55	156.61	588.92	83.30	238.21	68.29	166.48	151.56	420.39

shows the percentage of improvements in accuracy obtained by JacSim over all other methods with the Wikipedia dataset.

4.2.4. Impact of d on Accuracy of Embedding Methods

In this section, we analyze whether increasing the number of dimensions improves the effectiveness of embedding methods in computing the similarity of nodes. In Section 4.2.2, Figure 4 illustrates the accuracy of all embedding methods with the BlogCatalog, Cora, and DBLP datasets for different values of d; in addition, Figure 5 illustrates the results of the same experiments with the TREC and Wikipedia datasets. As observed in these figures, in some cases such as DWNS and NetMF with the BlogCatalog dataset, increasing the number of dimensions improves the accuracy of the embedding methods (i.e., refer to Figure 4); on the contrary, in some cases such as Line with the Wikipedia dataset (i.e., refer to Figure 5) and graphGAN with the BlogCatalog dataset (i.e., refer to Figure 4), increasing the number of dimensions adversely affects the accuracy of the embedding methods. In addition, in some cases such as DeepWalk with the DBLP dataset (i.e., refer to Figure 4) and node2vec with the TREC dataset (i.e., refer to Figure 5), we observe both improvement and reduction in accuracy by increasing the number of dimensions. In summary, increasing the value of d does not help improve the accuracy of embedding methods in computing the similarity of nodes. In Section 4.2.2, Table 3 indicates the value of d showing the best accuracy for each embedding method with our datasets.

As represented in Table 3, some embedding methods show their best accuracies when $d=64$ or $d=512$; we call such a case a suspicious one since it may possible to improve the accuracy of the embedding method by assigning a lower (i.e., 32) or higher (i.e., 1024) value to d, respectively. However, if the accuracy of a suspicious case is not comparable with the accuracy of the best method in the dataset, conducting the aforementioned experiment is not beneficial since our overall observations will not be affected by the new result; for example, although ATP with the BlogCatalog dataset shows its highest accuracy when $d=512$ as a suspicious case, its accuracy is quite lower than that of NetMF as the best method for the same dataset (refer to Figure 6). In Table 3, there are only four following real suspicious cases that we need to consider them: both DeepWalk and Line show their highest accuracies when $d=512$ with the Cora dataset and their accuracies are very close to that of JPRank as the best method with Cora (i.e., refer to Figure 7). In addition, DeepWalk and NetMF show their highest accuracies when $d=64$ and $d=512$ with the TREC dataset, respectively; their accuracies are very close to that of SimRank* as the best method with TREC (i.e., refer to Figure 9). Therefore, we conduct the following four new experiments:

• DeepWalk with the Cora dataset and $d=1024$
• Line with the Cora dataset and $d=1024$
• DeepWalk with the TREC dataset and $d=32$
• NetMF with the TREC dataset and $d=1024$

Table 14. Results of four extra experiments for suspicious cases.

		MAP	Precision	Recall	PRES	F-Score
DeepWalk with Cora dataset	$d=512$	0.023	0.398	0.031	0.026	0.053
	d= 1024	0.023	0.398	0.031	0.026	0.053
Line with Cora dataset	$d=512$	0.023	0.391	0.032	0.026	0.053
	$\mathit{d}\mathbf{=}\mathbf{1024}$	0.023	0.391	0.031	0.026	0.053
DeepWalk with TREC dataset	$\mathit{d}\mathbf{=}\mathbf{32}$	0.022	0.025	0.039	0.027	0.028
	$d=64$	0.026	0.033	0.054	0.035	0.038
NetMF with TREC dataset	$d=512$	0.025	0.034	0.054	0.036	0.038
	$\mathit{d}\mathbf{=}\mathbf{1024}$	0.025	0.033	0.053	0.035	0.038

represents the accuracies of the four new experiments (i.e., in bold face) along with the accuracies of their corresponding suspicious cases. In case 1, DeepWalk shows the same accuracy as it does when $d=512$; in addition, in cases 2 and 4, Line and NetMF show similar accuracies as they do when $d=512$, respectively. In case 3, DeepWalk shows lower accuracy in comparison with $d=64$. Therefore, we do not need to apply any changes in our results represented in Table 3.

4.2.5. Efficiency Evaluation

In this section, we carefully analyze the efficiency (i.e., execution time) of embedding methods in computing the similarity of nodes and compare it with that of similarity measures.

• Link-Based Similarity Measures

In order to conduct a fair comparison, we implemented the matrix form of JacSim, JPRank, SimRank*, and SimRank without applying any acceleration techniques such as multi-processing (as the simplest technique), fine-grained memorization [9], partial sums memoization [49], and backward local push and Monte Carlo sampling [50]. Since the execution time could slightly change depending on the system resources such as CPU overload, to obtain an accurate execution time, we run each similarity measure on eight iterations for five times with a dataset and the average run time over the five executions is regarded as the final execution time of the similarity measure. Note that we consider only the elapsed time to compute the similarity scores as the execution time; the required time to store the results of similarity computation in a file or a database is not considered.

Table 15. Execution time (minutes) of all similarity measures with our datasets.

	JacSim	JPRank	SimRank*	SimRank
BlogCatalog	457.24	-	2.02	4.38
Cora	3.54	8.22	0.21	0.32
DBLP	3.85	8.41	0.51	0.70
TREC	39.80	91.17	1.45	5.10
Wikipedia	55.29	-	0.28	0.53

shows the execution time (minutes) of similarity measures with our five datasets (As already explained in Section 4.2.1, we do not apply JPRank to undirected graphs BlogCatalog and Wikipedia). With all datasets, SimRank* shows the best efficiency since it requires only one matrix multiplication in Equation (A8). With undirected datasets (i.e., BlogCatalog and Wikipedia), JacSim shows the worst efficiency since it requires two matrix multiplications and a pairwise normalization paradigm to compute matrix E in Equation (A6). With directed datasets (i.e., Cora, DBLP, and TREC), JPRank shows the worst efficiency since it requires four matrix multiplications and two pairwise normalization paradigms to compute matrices E and $E^{\prime }$ in Equation (A10). However, among all the available cases in Table 15, JacSim with the BlogCatalog dataset shows the worst efficiency, although BlogCatalog has less nodes than Cora, DBLP, and TREC datasets. The reason is that there are “32,787,165” node-pairs with non-empty common in-link sets in this dataset, which makes the calculation of matrix E expensive; the number of these node-pairs in Cora, DBLP, TREC, and Wikipedia datasets are “229,306”, “466,990”, “1,391,293”, and “11,015,803”, respectively.

• Graph Embedding Methods

For embedding methods, the execution time is regarded as the summation of a learning time (i.e., elapsed time to construct low-dimensional representation vectors) and a similarity computation time (i.e., elapsed time to compute the similarity scores of all pairs of representation vectors by employing Cosine). With each dataset, the learning time of an embedding method is regarded as the average run time over the five executions of the method. We implemented Cosine based on a matrix/vector multiplication technique, which is significantly (i.e., almost 30 times) faster than its conventional implementation. In the case of similarity computation time, we consider only the elapsed time to compute the similarity scores by applying Cosine; the required time to store the results in a file or a database is not considered as we did for similarity measures.

Table 16. Learning time of all embedding methods for different values of d with all datasets.

		ATP	BoostNE	DeepWalk	DWNS	graphGAN	Line	NERD	NetMF	node2vec
BlogCatalog	$d=64$	3.69	3.44	41.25	30.38	151.37	6.53	9.10	0.80	19.62
	$d=128$	4.21	3.73	53.16	52.86	308.36	10.49	11.12	1.12	24.16
	$d=256$	4.88	4.22	72.91	71.50	397.20	23.03	14.02	1.85	25.15
	$d=512$	6.21	4.63	106.40	90.26	579.83	36.19	30.30	3.31	26.29
Cora	$d=64$	5.05	2.12	10.21	85.67	-	7.65	4.73	5.64	0.26
	$d=128$	5.24	2.54	12.22	116.03	-	10.01	6.09	7.76	0.31
	$d=256$	7.44	3.98	16.82	175.93	-	18.94	13.70	9.15	0.35
	$d=512$	11.30	9.44	24.22	216.82	-	34.62	25.94	12.38	0.61
DBLP	$d=64$	5.85	2.42	4.88	89.76	891.96	6.55	6.01	5.63	0.17
	$d=128$	6.53	2.86	6.46	101.19	1009.23	10.24	8.01	6.56	0.22
	$d=256$	7.67	3.32	8.64	150.27	1479.23	19.37	13.61	7.33	0.26
	$d=512$	12.18	5.97	12.43	175.10	2292.45	35.67	26.87	9.17	0.38
TREC	$d=64$	52.23	5.02	61.65	369.48	-	7.06	5.45	10.88	1.36
	$d=128$	57.86	5.84	79.98	408.59	-	10.10	7.92	11.55	1.65
	$d=256$	63.69	8.19	105.36	479.09	-	20.77	16.45	17.75	1.96
	$d=512$	76.22	9.90	144.88	639.21	-	33.98	27.36	26.46	2.41
Wikipedia	$d=64$	1.03	1.00	16.20	9.73	29.55	6.65	8.88	0.26	4.92
	$d=128$	1.05	1.05	18.39	11.92	32.81	11.28	10.68	0.30	5.32
	$d=256$	1.33	1.15	25.77	18.32	49.52	18.61	17.04	0.40	5.55
	$d=512$	1.85	1.78	38.35	27.6	92.38	36.66	28.35	0.80	5.58

represents the learning time (minutes) of all embedding methods for different values of d with all datasets where bold face numbers indicate the best efficiency with each value of d in a dataset. As observed in the table, NetMF shows the best efficiency among all embedding methods with the BlogCatalog and Wikipedia datasets regardless of the value of d, node2vec shows the best efficiency among all methods with Cora, DBLP, and TREC datasets regardless of the value of d, BoostNE, and ATP almost have better efficiency after NetMF and node2vec with all datasets, and graphGAN shows the worst efficiency among all embedding methods with all datasets regardless of the value of d.

Table 17. Similarity computation time (minutes) with different values of d for all datasets.

	BlogCatalog	Cora	DBLP	TREC	Wikipedia
$d=64$	4.07	21.05	17.52	72.89	0.90
$d=128$	4.51	22.45	19.87	74.07	0.92
$d=256$	4.91	23.28	20.44	76.42	1.10
$d=512$	6.13	25.69	23.23	83.06	1.62

shows the similarity computation time (minutes) based on different vector sizes (i.e., $d=64,128,256,512$) with our five datasets. Note that the similarity computation time for a dataset depends on the number of node-pairs and the representation vector’s size (i.e., the value of d); for example, with the BlogCatalog dataset, the required time to compute Cosine for all node-pairs with vector size 64 obtained by any embedding methods (except APT and NERD) is 4.07. In the case of ATP and NERD with any value of d, the similarity computation time in Table 17 is multiplied by two since these methods construct two vectors for each node (i.e., target and source vectors) where we apply Cosine to the corresponding target vectors and source vectors of a node-pair separately; finally, the highest score is regarded as the final similarity score of the node-pair.

In order to easily compare the efficiency of all embedding methods at a glance, Figure 11 illustrates their execution times (i.e., the summation of the learning time and the similarity computation time) with all datasets; we excluded graphGAN since its execution time value is quite larger than other embedding methods. For example, with the BlogCatalog dataset when $d=64$, the execution time of ATP is 11.83 as the summation of 3.69 (i.e., the learning time from Table 16) and $2\times 4.07$ (As explained before, for simplicity, we regard the similarity computation time as twice that in Table 17; for ATP with BlogCatalog when $d=64$, the real Cosine calculation time is 8.33 (≃$2\times 4.07$).) (i.e., twice the similarity computation time from Table 17); in addition, the execution time by DeepWalk is 45.32 as the summation of 41.25 (i.e., the learning time) and 4.07 (i.e., the similarity computation time).

• Efficiency Comparison

In order to make a meaningful comparison, for each of our five datasets, we compare the efficiency of the best embedding method with that of the best similarity measure from Section 4.2.3, as follows:

• BlogCatalog: as observed in Figure 6, NetMF with $d=256$ (i.e., refer to Table 3) and SimRank* are the best embedding method and similarity measure showing highest accuracy, respectively. The execution time of NetMF with $d=256$ is 6.76 (i.e., 1.85 from Table 16 plus 4.91 from Table 17), while the execution time of SimRank* is 4.38 (refer to Table 15); SimRank* shows almost 35% better efficiency than NetMF.
• Cora: as observed in Figure 7, Line (i.e., with $d=512$) and JPRank are the best embedding method and similarity measure, respectively. The execution time of Line when $d=512$ is 60.31 (i.e., 34.62 + 25.69), while the execution time of JPRank with this dataset is 8.22; JPRank is almost 7.3 times more efficient than Line.
• DBLP: as observed in Figure 8, DeepWalk (i.e., with $d=128$) and JPRank show the highest accuracies among embedding methods and similarity measures, respectively. The execution time of DeepWalk when $d=128$ is 26.33 (i.e., 6.46 + 19.87) and that of JPRank is 8.41, which means that JPRank is 3.1 times more efficient than DeepWalk.
• TREC: as observed in Figure 9, NetMF with $d=512$ and SimRank* are the best embedding method and similarity measure, respectively. The execution time of the former method is 109.52 (i.e., 26.46 + 83.06) and that of the latter one is only 1.45, which means SimRank* is significantly faster than NetMF.
• Wikipedia: as observed in Figure 10, DeepWalk with $d=64$ and JacSim show the best accuracies among embedding methods and similarity measures, respectively. The execution time of DeepWalk when $d=64$ is 17.10 (i.e., 16.20 + 0.90) and the execution time of JacSim is 55.29, which means DeepWalk is almost 3.2 times faster than JacSim.

4.2.6. Discussion

Based on the results of our extensive experiments with embedding methods and similarity measures in Section 4.2.3 and Section 4.2.5, we observe that the latter technique is better than the former one to compute the similarity of nodes in graphs for the following reasons.

• First, similarity measures outperform embedding methods with all datasets in terms of effectiveness except with the BlogCatalog dataset where NetMF (i.e., with $d=256$) shows the better accuracy than SimRank*; however, its efficiency is 54% less than that of SimRank* with this dataset.
• Second, similarity measures show better efficiency with all datasets except with the Wikipedia dataset where DeepWalk (i.e., with $d64$) is almost 3.2 times faster than JacSim; however, for this dataset, JacSim shows better effectiveness than DeepWalk and significantly outperforms it in terms of all five evaluation metrics as observed in Table 13.
• Third, similarity measures have a very low number of parameters than embedding methods, thereby leading to a simpler parameter tuning process to possibly obtain a better accuracy; for example, JPRank has only three parameters as ${\alpha }_1$, ${\alpha }_2$, and $\beta$ in Equation (A10), while DeepWalk has six parameters as the window size, walk length, number of dimensions, number of walks, size of training data, and learning tare.

In addition to the above findings, we observed that DeepWalk and its variants (i.e., Line and NetMF) show better effectiveness than other embedding methods in the task of similarity computation of nodes in graphs. Furthermore, it is shown that increasing the value of d (i.e., number of dimensions) does not help improve the accuracy of embedding methods in computing the similarity of nodes.

5. Conclusions

Embedding methods aim to represent each node in a given graph as a low-dimensional vector while preserving the neighborhood similarity, semantic information, and community structure of the nodes in the original graph. The dimensions in the low-dimensional vectors can be interpreted as latent features and the obtained vectors can be employed to compute the similarity of nodes in the graph. In this paper, we evaluated and compared both the effectiveness and efficiency of embedding methods in the task of computing similarity of nodes in graphs with those of link-based similarity measures by conducting extensive experiments with five datasets. We observed the following findings based on the results of our experiments. The similarity measures outperform embedding methods in terms of effectiveness with all datasets except with the BlogCatalog dataset where DeepWalk and NetMF show the best accuracy. The similarity measures are more efficient than embedding methods in similarity computation of nodes with all datasets except with the Wikipedia dataset where DeepWalk shows better efficiency. Finally, similarity measures are better to compute the similarity of nodes in graphs.