Complex Knowledge Graph Embeddings Based on Convolution and Translation

Abstract

Link prediction involves the use of entities and relations that already exist in a knowledge graph to reason about missing entities or relations. Different approaches have been proposed to date for performing this task. This paper proposes a combined use of the translation-based approach with the Convolutional Neural Network (CNN)-based approach, resulting in a novel model, called ConCMH. In the proposed model, first, entities and relations are embedded into the complex space, followed by a vector multiplication of entity embeddings and relational embeddings and taking the real part of the results to generate a feature matrix of their interaction. Next, a 2D convolution is used to extract features from this matrix and generate feature maps. Finally, the feature vectors are transformed into predicted entity embeddings by obtaining the inner product of the feature mapping and the entity embedding matrix. The proposed ConCMH model is compared against state-of-the-art models on the four most commonly used benchmark datasets and the obtained experimental results confirm its superiority in the majority of cases.

1. Introduction

Knowledge graphs use points and edges to represent individuals in the real world and their relations to other individuals. A typical knowledge graph contains millions of individuals and hundreds of millions of relations. Knowledge graphs have been widely used in intelligent applications such as search engines, question-and-answer systems [1], and recommendation systems [2]. Since most knowledge graphs are either constructed collaboratively or (partially) automatically, they often suffer from incompleteness. For example, in Freebase [3] and DBpedia [4], the place of birth is missing in more than 66% of person entries. Moreover, in DBpedia, more than 58% of scientists are not linked to the predicate that describes what they are known for, [5]. Therefore, many studies have focused on the knowledge graph complementation task. Link prediction, which is the prediction of missing triplets based on the existing complete triplets in a knowledge graph, is the main method to overcome the incompleteness of the knowledge graph. Figure 1 shows an example of link prediction, where a solid line represents an existing relationship and a dashed line represents a relationship that needs to be inferred. For example, the relational link prediction (Michelle LaVaughn Obama, ?, Female) can be inferred from (Barack Hussein Obama, hasGender, male) and (Barack Hussein Obama, isMarriedTo, Michelle LaVaughn Obama).

Knowledge graph embedding (KGE) models have been particularly successful in solving link prediction tasks [7]. Also known as Knowledge Representation Learning (KRL), KGE is inspired by the way the human brain thinks about problems and symbolizes knowledge to perform more complex reasoning and thinking by acquiring semantic information about knowledge. It is an upstream task in machine reading comprehension, intelligent question and answer generation, information retrieval, medical treatment, etc. The superior performance of KGE models can greatly improve the accuracy of generated intelligent questions and answers and can make the search engines more intelligent in returning more proper answers that people need.

Many existing KGE models obtain vector representations of entities and relations by embedding them into a low-dimensional vector space and performing complex mathematical operations on the vector representations to measure their plausibility in the real world. TransE [8] is the most classical KGE model, which embeds entities and relations into Euclidean space and measures the rationality of a triplet by computing the distance between the sum of the head entity vector $h$ and the relation vector $r$, and the tail entity vector $t$ in the Euclidean space. If the triplet is correct, the distance between $(h+r)$ and $t$ should be the smallest. TransE is simple and interpretable but cannot handle complex mapping patterns. In order to solve this problem, TransH [9] added the head entity to the relationship to generate their interaction features. More specifically, the head entity $h$ and tail entity $t$ are projected onto the hyperplane before the interaction to get the projection vector of the head entity $h_p$ and the projection vector of the tail entity $t_p$, such that, using the similar semantic information vector similarity principle, $h_p+r\approx t_p$ is made to hold as much as possible. This translation-based type of model, however, has limited depth and can only learn shallow features of the knowledge graph. Improving the learning ability of the model without changing the model structure can only increase the number of shallow features, which results in an inability to apply the model to large knowledge graphs. Subsequently, a Convolutional Neural Network (CNN) approach was introduced for improving the shortcomings of the depth of the shallow models. ConvE [10] was the first embedding model using a 2D convolution for knowledge graph complementation. It first embeds entities and relations into a low-dimensional vector space, then concatenates entity embeddings and relation embeddings into a 2D matrix, and finally, extracts features from the matrix using 2D-convolution operations. However, this feature extraction is not very meaningful because the entities and relations are simply concatenated together without sufficient interaction. The internal elements of the entities and relations do not interact, and only a small number of elements at the connection points can interact locally. Thus, the model does not learn the deep semantic features and wastes the powerful learning ability of CNN. Therefore, in this paper, we introduce a new KGE model (ConCMH), based on a 2D convolution, which uses multiplication and inner product operations of matrices in the complex space for interaction. The innovation of ConCMH lies in the abandonment of the traditional Euclidean space and embedding entities and relationships into the complex space instead. Consisting of a real part and an imaginary part, the vector representation in the complex space can show different characteristics of entities and relationships from multiple aspects. Previously proposed KGE models all use a single interaction. For instance, ConvE simply concatenates entities and relationships, InteractE cross-concatenates the elements of entities and relationships, and TransH adds the head entity to the relationship to generate their interaction features. In contrast, the proposed ConCMH model uses a mixed interaction of addition and multiplication to further deepen the semantic connection between the head entity and relationship. Previous CNN-based models, such as ConvE, ConvKB, and InteractE, ultimately project the interaction features of the head entity and relationship into the entity embedding space through a multiplication operation to match all candidate entities. Working in the complex space, ConCMH replaces the traditional multiplication operation with an inner product operation, making the interaction features of the head entity and relationship closer to the correct tail entity.

The main contributions of this paper can be summarized as follows:

• A novel link-prediction model of a combined translation-based and CNN-based type is proposed, utilizing a mapping relationship $h\times r\approx t$ in the complex space, which allows it to achieve more adequate entity embedding and relationship embedding of the interactions at the element level and explore the interactions from many different perspectives instead of being bound to a single additive or multiplicative interactions. In addition, the vector representation in the complex space allows the approach to take into account different properties of entities from many different aspects.
• The impact of the “test leakage” problem on model performance is numerically quantified and a practical way for reducing its effect is demonstrated.

2. Related Work

In recent years, many KGE models have been proposed, such as translation-based, CNN-based, geometry-based, path-based, etc. In this paper, we focus only on the former two types and propose a combined use of these in our ConCMH model, described in Section 4. Compared to the CNN-based models, the translation-based models have a simpler structure and use fewer parameters; these models only learn shallow features of a knowledge graph in order to be able to extend their operation to large knowledge graphs. In contrast, CNN-based models possess the ability to mine deep semantic information.

2.1. Translation-Based KGE Models

As a pioneering work in KGE, the TransE model paved the way for a variety of translation-based models. TransE embeds the triplet into a low-dimensional vector space and the relation is characterized as a translation operation between the head entity vector and the tail entity vector. The model is very efficient while also achieving very good link prediction performance. However, TransE encounters difficulties when dealing with complex relationships such as one-to-many, many-to-one, many-to-many, and symmetric relationships. For example, when dealing with the triplets (Obama, President, United States) and (Trump, President, United States), the TransE model identifies the head entity “Obama” and the head entity “Trump” as equal. However, in the real world, Obama and Trump seem to have nothing in common except that they have both served as Presidents of the United States. To solve this transfer problem, Wang et al. [9] proposed the TransH model, which introduced the concept of a hyperplane and obtained different entity representations by projecting entities onto the hyperplane.

There are also tensor-decomposition-based models similar to the translation-based models, which also use a simple structure in exchange for high efficiency. RESCAL [11] is a typical representative of this subgroup. It encodes each triplet as a tensor and a matrix and handles the knowledge graph complementation problem by optimizing the bilinear product between the core tensor and the factor matrix. Although RESCAL is a powerful model, it can easily overfit during training due to the large number of parameters used. DistMult [12] reduces the number of operations in RESCAL by replacing the core tensor with a diagonal matrix, which reduces the possibility of overfitting. However, DistMult cannot encode antisymmetric relations, which is its fatal flaw. In order to solve this problem, ComplEx [13] embeds DistMult into the complex vector space and encodes the antisymmetric relations in the triplet through the anti-symmetry of the Hermitian inner product, which makes up for the deficiencies of DistMult.

2.2. CNN-Based KGE Models

Dettmers et al. [10] argue that previous models using simple structures and few parameters were shallow and fast models, designed to scale to large knowledge graphs. These models, however, do not learn expressively as deep multilayer models, and their simple structure and efficient operations potentially limit their performance. To this end, Dettmers et al. [10] proposed ConvE as a multilayer CNN model for link prediction. ConvE uses a 2D convolution and fully connected layers to handle the interactions between entities and relations, which allows it to outperform previous models. In addition, the model has high parameter efficiency, yielding the same performance as DistMult [12] and R-GCN [14] with 8-fold and 17-fold parameter reduction, respectively. Inspired by [10], Nguyen et al. [15] also realized that CNNs can capture the effective features of interactions between triplets. Based on ConvE, the input to the CNN in their model, called ConvKB, was changed to a concatenation of head entities, relations, and tail entities, and multiple filter operations were utilized in order to generate different feature mappings, which improved the model performance to some extent. Vashishth et al. [16] affirmed the application of convolutional filters to the 2D reconstruction of entity and relational embeddings, but they found that the number of interactions that can be captured by ConvE and ConvKB is limited. The limited number of interactions largely programmatically limits the CNN feature-extraction capability. Therefore, Vashishth et al. proposed the InteractE model [16] utilizing a feature replacement and a circular convolution, which alleviated this deficiency to some extent.

Table 1. Comparison of the KGE models.

Model	Scoring Function	Relational Parameters	Space Complexity
TransE	$f_r(h,t)=\left|\right|h^e+r^e-t^e|{|}_p$	$r^e\in R^{d_r}$	$o(n_e{d}_e+n_r{d}_r)$
TransH	$f_r(h,t)=\left|\right|h_p^e+d_r^e-t_p^e|{|}_p$	$r^e\in R^{d_{2r}}$	$o(n_e{d}_e+n_r{d}_{2r})$
RESCAL	$f_r(h,t)=h^e\cdot w_r^e\cdot t^e$	$r^e\in R^{d_e^2}$	$(n_e{d}_e+n_r{d}_r^2)$
DistMult	$f_r(h,t)=\langle h^e,r^e,t^e\rangle$	$r^e\in R^{d_e}$	$o(n_e{d}_e+n_r{d}_r)$
ComplEx	$f_r(h,t)=\mathrm{R}\mathrm{e}(\langle h^e,r^e,\stackrel{-}{t^e}\rangle )$	$r^e\in R^{d_e}$	$o(n_e{d}_e+n_r{d}_r)$
ConvE	$f_r(h,t)=f(vec(f([\stackrel{-}{h^e};\stackrel{-}{r^e}]\ast \omega ))W)t^e$	$r^e\in R^{d_r}$	$o(n_e{d}_e+n_r{d}_r)$
InteractE	$f_r(h,t)=f\left(vec\right(f\left(\varphi \right(P_k)\ast \omega )\left)W\right)t^e$	$r^e\in R^{d_r}$	$o(n_e{d}_e+n_r{d}_r)$
ConvKB	$f_r(h,t)=f\left(vec\right(f\left(\right[h^e;r^e;t^e]\ast \omega )\left)W\right)$	$r^e\in R^{d_r}$	$o(n_e{d}_e+n_r{d}_r)$
ConCMH (our model)	$f_r(h,t)=\langle conv(h^e,r^e),t^e\rangle$	$r^e\in R^{d_{2r}}$	$o(n_e{d}_{2e}+n_r{d}_{2r})$

summarizes the scoring functions, relational parameters, and space complexity of the described KGE models, including also the ConCMH model proposed in this paper, where $d_e$ and $d_r$ denote the entity embedding dimension and relational embedding dimension, respectively; $n_e$ and $n_r$ denote the number of entities and relations, respectively; $\ast$ denotes the convolution operation; $\stackrel{-}{t^e}$ denotes the conjugate of the tail entity embedding; $\stackrel{-}{h^e}$ and $\stackrel{-}{r^e}$ denote the tandem operation of the head entity embedding and relational embedding, respectively; $[h^e;r^e;t^e]$ denotes the tandem operation of head entity embedding, relational embedding, and tail entity embedding; $Re\left(\right)$ denotes the real part of the result; $f\left(\right)$ denotes the nonlinear function; and $\langle \rangle$ denotes the inner product operation.

3. Background

3.1. Knowledge Graph

A knowledge graph can be represented by $G=\{E,R,T\}$, where $E$ denotes the set of entities, $R$ denotes the set of relations, and $T$ denotes the set of triplets $\langle h,r,t\rangle$, each consisting of a head entity $h$ and a tail entity $t$, connected by a directed edge representing an $r$ relationship.

3.2. Link Prediction

The goal of the link prediction task is to use the complete triplets that already exist in a knowledge graph to infer the missing triplets. Depending on the missing part, the link prediction is further divided into: (i) entity prediction, i.e., given $\langle h,r,?\rangle$ to reason about the missing tail entity, or given $\langle ?,r,t\rangle$ to predict the missing head entity; and (ii) relation prediction, i.e., given $\langle h,?,t\rangle$ to reason about the missing relation using the head entity and tail entity.

4. Proposed Model

Inspired by ConEx [17] and CTKGC [18] models, we named the proposed model ConCMH.

ConCMH includes an embedding layer, interaction layer, feature extraction layer, and output layer, as shown in Figure 2. At the embedding layer, the head entity and the relation are embedded into the complex space $C$ to obtain the corresponding head entity embedding and relation embedding. At the interaction layer, the interaction between the head entity and the relation is made at the element level, whereby $h^e\times r^e$ is made in the complex space $C$ and the real part of $h^e\times r^e$ is taken out to form the entity–relation interaction matrix. At the feature extraction layer, the entity–relation interaction matrix is subject to feature extraction to obtain different feature mappings, which are then transformed into vectors by a remodeling operation. At the output layer, the inner product of feature vectors and entity embedding matrix is made in the complex space $C$ to match the feature vector with all candidate entity embedding. Finally, scoring is performed with a Sigmoid function, and the score of each candidate entity is obtained.

The ConCMH layers are described in more detail in the following sections.

4.1. Embedding Layer

Knowledge graphs describe real-world facts in the form of triplets, which are not directly mathematical, so first, we have to embed entities and relations into the complex space in order to obtain vector representations of entities and relations. Given a head entity $h\in \{h_1,h_2,h_3,.....,h_{n_h}\}$ and a relation $r\in \{r_1,r_2,r_3,........,r_{n_r}\}$, we generate the corresponding real part $h\_re{l}^e$ and imaginary part $h\_i{m}^e$ of the head entity embedding, and the real part $r\_re{l}^e$ and imaginary part $r\_i{m}^e$ of the relation embedding, as follows:

$$\begin{array}{c}h\_re{l}^e=\{h\_re{l}_1^e,h\_re{l}_2^e,....h\_re{l}_{n_h}^e\}\in n_h\times d_e;\\ h\_i{m}^e=\{h\_i{m}_1^e,h\_i{m}_2^e,....h\_i{m}_{n_h}^e\}\in n_h\times d_e;\\ r\_re{l}^e=\{r\_re{l}_1^e,r\_re{l}_2^e,....r\_re{l}_{n_r}^e\}\in n_r\times d_r;\\ r\_i{m}^e=\{r\_i{m}_1^e,r\_i{m}_2^e,....r\_i{m}_{n_r}^e\}\in n_r\times d_r;\end{array}$$

(1)

where $n_e$ denotes the number of entities, $n_r$ denotes the number of relations, $n_h$ denotes the number of head entities, $d_e$ denotes the entity embedding dimension, and $d_r$ denotes the relation embedding dimension. Each head entity embedding $h^e$ is represented by $(h\_re{l}^e+h\_i{m}^e)$, and each relation embedding $r^e$ is represented by ($r\_re{l}^e+r\_i{m}^e)$, which are then sent to the interaction layer.

4.2. Interaction Layer

To establish a mapping relationship among head entity embedding, relation embedding, and tail entity embedding (so that the interaction feature between the head entity and relation is closer to the tail entity), we take the head entity embedding and relation embedding for a product operation in the complex space and then make the interaction between the real part $h\_re{l}^e$ and imaginary part $h\_i{m}^e$ of the head entity, and the real part $r\_re{l}^e$ and imaginary part $r\_i{m}^e$ of the relation embedding at the element level. The interaction matrix $h\_r$ is generated by taking the real part of the results, as follows:

$$h\_r=Re(h^e\times r^e)=h\_re{l}^e\times r\_re{l}^e-h\_i{m}^e\times r\_i{m}^e$$

(2)

Each element in the interaction matrix $h\_r$ is composed of head entity embedding and relation embedding, which reflects the full interaction between embeddings. Next, the interaction matrix is sent to the feature extraction layer, where effective features are extracted through a convolution operation to ensure that the model learns the deep semantic information.

4.3. Feature Extraction Layer

At the feature extraction layer, the interaction matrix is subject to feature extraction by a 2D convolution operation to obtain more effective features forming the feature mapping tensor $T\in C^{c\times m\times n}$, where $c$ is the number of feature mappings with dimensions $m$ and $n$. Then, tensor $T$ is remodeled into a vector by $vec\left(T\right)\in C^{cmn}$. Finally, vector $T$is sent to a fully connected network, and the feature vector is generated by the weight parameter $W$ and the offset parameter $b$, as follows:

$$conv\left(h^e,r^e\right)=f\left(vec\left(f\left(Re\left(h^e\times r^e\right)\right)\ast \omega \right)\cdot W+b\right)$$

(3)

where $\ast$ denotes the convolution operation, $\omega$ denotes the convolution kernel, the outermost $f(·)$ denotes the linear unit function (ReLU), and the inner $f(·)$ denotes the interactive operation of the head entity embedding $h^e$ and relation embedding $r^e$. To speed up the model convergence, we set the convolution kernel to 3 × 200, taking into account the experience obtained with the CTKGC model [18].

4.4. Output Layer

At the output layer, ConCMH divides the output of the feature extraction layer into two parts according to the size of the entity embedding dimension $d_e$ and relation embedding dimension $d_r$, which are used as the real part $rel$ and imaginary part $im$ of the feature vector, respectively. To reduce the training time of the model and improve its efficiency, the evaluation method of “K vs. N” is applied. The feature vector is projected into the $n_e\times d_e$ entity embedding space by a parameterized linear transformation of matrix $W\in C^{cmn\times d}$, where $c$ denotes the number of features output by the convolution operation, $m$ and $n$ denote the number of rows and columns of the feature mapping matrix, respectively, and $d$ denotes the sum of $d_e$ and $d_r$. In addition, the inner product of the feature vectors is obtained with all entities to generate tail entity prediction objects $t^p$. Finally, a Sigmoid function is used to calculate the score of each candidate triplet to find the candidate tail entity with the highest score. The scoring function of ConCMH is shown below:

$$\begin{gathered} ConCMH\left(h,r,t\right)=⟨conv\left(h^e,r^e\right),t^e⟩ \\ =rel\times E\_rel+rel\times E\_im+im\times E\_rel-im\times E\_im \end{gathered}$$

(4)

We use the Adam optimizer to train the proposed ConCMH model and optimize the loss generated by a given triplet $<h,r,t>$ by minimizing the following binary cross-entropy loss function $L$:

$$L(ConCMH,y)=-\frac{1}{n}\sum _{i=1}^n(y^{\left(i\right)}\mathrm{l}\mathrm{o}\mathrm{g}({\widehat{y}}^{\left(i\right)})+(1-y^{\left(i\right)})\mathrm{l}\mathrm{o}\mathrm{g}(1-{\widehat{y}}^{\left(i\right)})),$$

(5)

where $\widehat{y}$ denotes the vector of predicted scores and $y\in \left[\mathrm{0,1}\right]$ denotes the label vector. When $y=0$, the triplet $<h,r,t>$ is an incorrect triplet; and when $y=1$, the triplet $<h,r,t>$ is a correct triplet reflecting the real world.

5. Experiments and Results

5.1. Datasets

Four datasets, namely, WN18 [8], FB15k [8], the Unified Medical Language System (UMLS) [19], and YAGO3-10 [10], were considered for use in the experiments conducted for performance comparison of the proposed ConCMH model with the state-of-the-art (SOTA) models.

WN18 is a subset of Wordnet, a broad-coverage English lexical-semantic network aggregated by researchers at Princeton University’s Cognitive Science Laboratory. There are 151,442 triplets in this strict-hierarchy dataset, containing 40,943 entities and 18 relations (the majority of which are superlative and sublative relations). FB15k is a subset of Freebase, consisting of 14,951 entities and 1345 relations mainly describing facts about movies, actors, awards, sports, and sports teams. UMLS is dedicated to the medical field, whereby 135 entities mainly describe a wide variety of diseases and 46 relations correspond to the means of treatment of diseases. YAGO3-10, a subset of YAGO3, contains 123,182 entities and 37 relations mainly used to describe various attributes of people, such as height, gender, and occupation.

As mentioned in [20], due to the “test leakage” problem, a large number of test triplets can be obtained in WN18 and FB15k simply by reversing the triplets in the training set and, because of this, simple models can get good results. To avoid this drawback, a large number of triplets possessing inverse relations in the WN18 and FB15k training sets were removed , resulting in the optimized subsets WN18RR [10] and FB15k-237 [21], respectively, whose details are shown in

Table 2. Details of datasets considered for use in the experiments.

Dataset	Number of Entities	Number of Relations	Number of Triplets in Training Set	Number of Triplets in Validation Set	Number of Triplets in Test Set
WN18	40,943	18	141,442	5000	5000
WN18RR	40,943	11	86,835	3034	3134
FB15k	14,951	1345	483,142	50,000	59,071
FB15k-237	14,541	237	272,115	17,535	20,466
UMLS	135	46	5216	652	661
YAGO3-10	123,182	37	1,079,040	5000	5000

along with details of the other datasets considered for use in the experiments.

5.2. Evaluation Metrics

In the link prediction task, the model performance is generally measured by using four main metrics, namely, the mean reciprocal rank (MRR), Hits@10, Hits@3, and Hits@1, all having a range of values between 0 and 1. MRR refers to the inverse of the average ranking of all candidate entities in the test set, whereas Hits@X refers to the probability that the correct entity appears in the top X candidates. Higher values of MRR and Hits@X are indications of better model performance.

5.3. Experimental Setup

To lower the computation effort, a grid search algorithm was utilized to determine the range of hyperparameters. The optimal hyperparameter settings of CTKGC were used as follows: entity embedding dimension and relational embedding dimension d:{50,200,400}, learning rate Lr:{0.01,0.003,0.001,0.0005}, label smoothing ratio Ls:{0,0.1,0.2,0.3}, batch size Bs:{128,256,512,1000}, feature dropout DropOut1:{0.1,0.2,0.3}, and hidden layer dropout DropOut2:{0.1,0.2,0.3}. Drawing on the experience of CTKGC, in order to speed up the model convergence, the convolution kernel was set to 3 × 200. For obtaining the best possible results on each dataset, the following hyperparameter values were used: for FB15k-237—Lr = 0.001, Bs = 1000, DropOut1 = 0.2, and DropOut2 = 0.2; for WN18RR—Lr = 0.001, Bs = 1000, DropOut1 = 0.2, and DropOut2 = 0.5; for UMLS—Lr = 0.001, Bs = 128, DropOut1 = 0.2, and DropOut2 = 0.3; for YAGO3-10—Lr = 0.01, Bs = 1000, DropOut1 = 0.2, and DropOut2 = 0.3.

5.4. Link Prediction: Results and Analysis

The link prediction performance of the proposed ConCMH model was compared to that of SOTA models, based on the UMLS, YAGO3-10, FB15k-237, and WN18RR datasets. Because of the “test leakage” problem, described in Section 5.1, the link prediction experiments were carried out on the optimized subsets FB15k-237 and WN18RR (instead of the full datasets FB15k and WN18), which were constructed by deleting the inverse relations. The obtained results are shown in

Table 3. Link prediction performance of models on the UMLS dataset.

Model	Hits@10	Hits@3	Hits@1	MRR
TransE ♦ [8]	0.992	0.921	0.641	0.797
DistMult ♦ [12]	0.446	0.301	0.199	0.281
ComplEx ♦ [13]	0.925	0.783	0.657	0.743
ConvE ♦ [10]	0.990	0.967	0.898	0.935
ConvKB ♦ [15]	0.993	0.961	0.608	0.785
R-GCN ♦ [14]	0.325	0.208	0.126	0.204
KBGAT [22]	0.998	0.995	0.986	0.990
HypER ♣ [23]	0.984	0.957	0.822	0.894
CTKGC [18]	0.995	0.975	0.915	0.948
ConCMH (proposed model)	1.000	0.996	0.973	0.853

♣ Results taken from [18]. ♦ Results taken from [22].

,

Table 4. Link prediction performance of models on the YAGO3-10 dataset.

Model	Hits@10	Hits@3	Hits@1	MRR
DistMult ♥ [12]	0.683	0.590	0.466	0.543
ComplEx ♥ [13]	0.690	0.594	0.468	0.547
ConvE [10]	0.660	0.560	0.450	0.520
HypER [23]	0.678	0.580	0.455	0.533
RotatE [24]	0.670	0.550	0.402	0.495
TuckER ♥ [25]	0.609	0.476	0.331	0.427
ConEx [17]	0.696	0.601	0.474	0.553
MquadE [26]	0.689	0.592	0.449	0.536
InteractE [16]	0.687	—	0.462	0.541
ConCMH (proposed model)	0.708	0.581	0.525	0.517

♥ Results taken from [17].

,

Table 5. Link prediction performance of models on FB15k-237 dataset.

Model	Hits@10	Hits@3	Hits@1	MRR
TransE & [8]	0.465	—	—	0.294
DistMult * [12]	0.419	0.263	0.155	0.241
R-GCN+ [14]	0.417	0.264	0.151	0.248
ConvE * [10]	0.501	0.356	0.237	0.325
ComplEx ♥ [13]	0.536	0.384	0.253	0.348
A2N [27]	0.486	0.348	0.232	0.317
NKGE [28]	0.510	0.365	0.241	0.330
InteractE [16]	0.535	—	0.263	0.354
CTKGC [18]	0.517	0.372	0.252	0.340
StructurE [29]	0.546	0.390	0.252	0.351
KMAE [30]	0.502	0.358	0.240	0.326
JointE [31]	0.543	0.393	0.262	0.356
ConCMH (proposed model)	0.595	0.445	0.392	0.394

^& Results taken from [15]. * Results taken from [23]. ♥ Results taken from [17].

and

Table 6. Link prediction performance of models on the WN18RR dataset.

Model	Hits@10	Hits@3	Hits@1	MRR
TransE & [8]	0.501	—	—	0.226
DistMult * [12]	0.490	0.440	0.390	0.430
ConvE * [10]	0.520	0.440	0.400	0.430
ComplEx ♥ [13]	0.547	0.490	0.438	0.475
A2N [27]	0.510	0.460	0.420	0.450
NKGE [28]	0.526	0.465	0.421	0.450
InteractE [16]	0.528	—	0.430	0.463
CTKGC [18]	0.521	0.472	0.426	0.459
StructurE [29]	0.585	0.500	0.425	0.479
KMAE [30]	0.524	0.465	0.415	0.448
JointE [31]	0.537	0.483	0.438	0.471
ConCMH (proposed model)	0.547	0.506	0.490	0.443

^& Results taken from [15]. * Results taken from [23]. ♥ Results taken from [17].

(the best value achieved among the models for a particular metric is shown in bold). Except where noted, the results in these tables are taken from the indicated reference sources. Some data in Table 4, Table 5 and Table 6 are missing because these are not found in the corresponding reference source.

On the FB15k-237 dataset (Table 5), the proposed ConCMH model outperforms all SOTA models based on all evaluation metrics. On the UMLS dataset (Table 3), ConCMH outperforms all SOTA models in terms of Hits@10 (reaching the maximum possible value of 1) and Hits@3, but it takes second and fifth places according to Hits@1 and MRR, respectively. On the WN18RR dataset (Table 6), in terms of Hits@3 and Hits@1, the proposed ConCMH model is the leader among all models considered, but it takes second (shared) place according to Hits@10 and drops to ninth place according to MRR. On the YAGO3-10 dataset (Table 4), ConCMH is the leader according to Hits@10 and Hits@1, but it takes fifth and eight places according to Hits@3 and MRR, respectively.

The reason for the ConCMH’s average performance, according to some evaluation metrics, on the UMLS, WN18RR, and YAGO3-10 datasets is that it uses a mixed interaction of addition and multiplication to deepen the semantic connection between entities and relationships, generating interaction features with deep semantic information. In addition, the 2D convolution is good at learning subtle differences between interaction features but ignores the surface information of the knowledge graph. In our future research, we will explore how ConCMH can learn surface information in knowledge graphs through convolution operations.

Generally, the proposed ConCMH model is not only superior to the translation-based models, such as TransE, DistMult, and ComplEx, but compared to the CNN-based models, such as ConvE, ConvKB, and CTKGC, its performance is also much better. The reason for this is that ConCMH combines the advantages of these two types of models so that it can not only learn the shallow features of the triplet surface but also can learn the semantic information hidden in the depth of a knowledge graph thanks to the use of a CNN connected deep structure. The biggest difference between ConCMH and CTKGC is that the former embeds entities and relations in the complex space, whereas the latter embeds entities and relations in the Euclidean space. As noted in [10], the only way to improve the expressiveness of shallow models by increasing the number of features is to increase the embedding size. Compared with that in Euclidean space, the vector embedded in the complex space is composed of a real part and an imaginary part, which actually increases the embedding size of the model invisibly, so the real part and imaginary part can reflect the different attributes of an entity from different angles, making the entity rich in features. ConCMH uses convolution operators in an efficient and fast parameter manner to form a deep network, which lowers the computational burden caused by increasing the embedding size. At its interaction layer, ConCMH combines addition, which is common for the translation-based models, with multiplication interaction, which is common for the CNN-based models, which not only retains the principle of similar semantic vectors in the Euclidean space but also gives full play to the nature of parameter sharing of convolution operators, thus linking the entities and relations in the triplets more closely. In addition, the entity–relation interaction matrix is subject to real part operation Re (), and some irrelevant feature interactions are discarded, so that the performance and efficiency of the model can be further improved, making it applicable to large-scale knowledge graphs.

5.5. ”Test Leakage” Problem: Results and Analysis

Before the “test leakage” problem was found, the WN18 and FB15k datasets were used to evaluate the performance of the KGE models. In 2015, Toutanova and Chen explained for the first time that the inverse relations in WN18 and FB15k may lead to this problem. For example, if the test set contains a triplet $<s, hyponym, o>$ and the training set contains its inverse triplet $<o, hypernym, s>$, then the test triplet can be easily mapped to the training triplet. In cases such as this, a model only learns which relations are inversions of other relations, rather than learning the semantic information of the triplets.

To investigate the seriousness of the “test leakage” problem, we conducted link prediction experiments on the full datasets WN18 and FB15k and their optimized versions WN18RR and FB15k-237 by comparing the performance of the ConCMH and ConvE models on each pair of the complete and optimized version of a dataset. The obtained results are shown in Figure 3. For the proposed ConCMH model, MRR is the performance metric whose values on WN18 and WN18RR differ the most (constituting a difference of 0.505 points). On FB15k and FB15k-237, the difference between the MRR values obtained for ConCMH is smaller at 0.337 points. The main reason for this is that the composition of relations in WN18 is different from that in FB15k. In WN18, 94% of the training set constitutes inverse relations, whereas in FB15k the number of inverse relations is 13% lower, resulting in a smaller number of deleted FB15k triplets when forming the FB15k-237 dataset, and in less difference between the MRR values. For ConvE, Hits@1 is the performance metric whose values on WN18 and WN18RR differ the most (constituting a difference of 0.545 points). On FB15k and FB15k-237, the biggest difference (equal to 0.451 points) is observed in the Hits@3 values obtained for ConvE. Comparing the two models, leaving “test leakage” aside, ConCMH and ConvE show almost equal performance on the WN18 dataset, with the biggest difference in the metric values of only 0.014 points (for Hits@1). However, on the optimized WN18RR dataset, the performance of the two models is quite different, with the biggest difference reaching 0.100 points (for Hits@1) and increased differences for other metric values as well. This demonstrates the importance of reducing the negative effect of the “test leakage” problem in order to come up with results that are as close to reality as possible.

5.6. ConCMH Parameter Values: Results and Analysis

We also explored the influence of different parameter values on the performance of the proposed ConCMH model (based only on the WN18RR dataset). The obtained results are shown in

Table 7. Influence of different parameter values on the ConCMH performance.

Parameter		Hits@10	Hits@3	Hits@1	MRR
Convolution	3 × 200	0.547	0.506	0.490	0.443
kernel’s size	5 × 200	0.496	0.463	0.452	0.438
Convolution	Conv2d	0.547	0.506	0.490	0.443
type	Conv1d	0.535	0.496	0.484	0.435

(the best value achieved among parameter values for a particular metric is shown in bold).

First, we experimented with using different sizes of the convolution kernel. When this size is expanded from 3 × 200 to 5 × 200, the values of all evaluation metrics decrease (to different degrees). So, the larger the convolution kernel’s size, the more subtle information ignored during feature extraction.

Then, we compared the use of two convolution types for feature extraction, namely 2D convolution (e.g., used by ConvE and CTKGC) and 1D convolution (e.g., used by ConvKB). In terms of computational nature, there is no difference between the two types; both move along the dimension of input data. However, the 1D convolution moves along a single dimension, while the 2D convolution moves along two dimensions. For the complementation of knowledge graphs, the essential difference between the two convolution types is related to the remodeling of input data. With the 1D convolution, the entity–relation interaction matrix needs to be remodeled into 2D, whereas with the 2D convolution, it is remodeled into 3D. In the case of the same embedding dimension, the 3D interaction matrix has more interaction points than the 2D interaction matrix, which deepens the connection of a triplet in its neighborhood, as illustrated in Figure 4. As can be seen from Table 7, when using 2D convolution, the values of all evaluation metrics for ConCMH are higher than those of 1D convolution.

6. Conclusions and Future Work

This paper has proposed a novel link-prediction model, called ConCMH, constructed by applying a new mapping relation h × r ≈ t, along with fully interacted entity embedding and relation embedding, in the complex space. The real part of this interaction forms an entity–relation interaction matrix, from where features are extracted by means of a 2D convolution. In addition, the feature mapping is matched with the prediction entity embedding through the inner product in the complex space. Compared to the state-of-the-art (SOTA) models considered, the ConCMH model features a simpler structure and demonstrates a more powerful link-prediction ability in the majority of cases, based on experiments conducted on four popular datasets. In addition, the proposed model can be easily expanded to larger and more complex knowledge graphs.

A disadvantage of ConCMH is that, similarly to some SOTA models, it only focuses on a triplet and ignores some special natures of the triplet itself. Many triplet entities and relations have special natures such as transitivity and generalization, which can be taken into account only by combining multiple triplets together. In our future work, we are going to study this issue in depth. In addition, compared with the Convolutional Neural Networks (CNNs) used in the field of computer vision, the CNN utilized by the proposed model is still too shallow. Thus, in the future, we plan to investigate the use of a combination of a variety of CNNs for increasing network depth.

Compared with the SOTA contrastive-learning-enhanced knowledge graph representation methods [32,33], ConCMH uses a scoring function to score triplets, ignoring the semantic similarity between related entities and related entity–relation pairs between different triplets. The interaction between the target entity and relationships is too one-sided. Other entities sharing the same relationship with the target entity may also have some semantic interaction with that target entity. In our future research, we will also take this aspect into consideration.