1. Introduction
Knowledge graphs, which represent knowledge from real world applications, contain abundant facts. In knowledge graphs, each fact is represented by a triple
which indicates that the relation
r between the head entity
h and tail entity
t. Knowledge graphs have been applied to various tasks such as explainable recommendation system [
1], question answering [
2] and prediction of future research collaborations [
3].
Predicting missing facts (i.e., link prediction) is a fundamental task in knowledge graph research. Various models aiming at embedding entities and relations into low-dimension spaces have been proposed. For example, TransE [
4] learned the embeddings of entities and relations by transforming head entity to tail entity according to the relation; RotatE [
5] and QuatE [
6] learned the embeddings of entities and relations by considering relations as rotations from head entities to tail entities. However, existing transformation based models fail to capture multiple relations between head and tail entities. For example, as shown in
Figure 1,
David Lynch is the director, the creator and an actor in the film
Mulholland Drive, i.e., there are three relations: directed, created and actedIn between
David Lynch and
Mulholland Drive. These relations between head entity
David Lynch and tail entity
Mulholland Drive have no semantic connections with each other, which should be represented by spatially dispersed embeddings. Most existing transformation based models, however, assume that there is only one relation between each pair of head and tail entities. For instance, for each triple
, their corresponding embeddings are assumed to be satisfied with
in TransE, which indicates, for
,
, and
, the embeddings of
,
,
are similar, as shown in
Figure 2c (i.e.,
). To overcome this challenge, we propose a novel approach that considers multiple relations between head and tail entities in knowledge graph.
In this paper, we propose a model called
DualQuatE which utilizes the various combinations of distinct rotations and translations to represent multiple relations between head and tail entities. Based on this, easily to think of RotatE combined with TransE in complex space and real space. However, it is hard to find a uniform mathematical expression to convey their combination. Therefore, we propose
DualQuatE which embeds entities and relations into dual quaternion space to combine rotation and translation. The dual quaternion consists of real part and dual part. More concretely, we embed entities with pure quaternions vectors in three-dimensional space to represent entity embeddings. To distinguish various relations between head entity
h and tail entity
t, we design a score function to utilize dual quaternion Hamilton product to model relations as interaction of rotation and translation. We utilize distinct interactions of rotations and translations to represent various relations between head and tail entities. Compared with RotatE and TransE in two-dimensional space, the dual quaternions space is eight-dimensional with six real degrees of freedom, three for translation and three for rotation; we can explore the interaction of rotation and translation with more free degrees in higher dimensions. Summarized in
Table 1, our model has rich expression abilities of relations (i.e., relation patterns and multiple relations).
To conclude, the contributions of our proposed model are listed as follows:
We introduce dual quaternions to knowledge graph embeddings.
We propose a novel transformation based model DualQuatE to overcome the challenge of multiple relations between two entities.
Our experiments denote that DualQuatE is effective compared to the existing state-of-the-art models.
The rest paper is organized as follows. In
Section 2, we introduce the related work.
Section 3 presents prerequisite knowledge about dual quaternions. In
Section 4, we describe our model. We present the results of experiments and make analysis and discussions in
Section 5. In
Section 6, we introduce the conclusion of this paper and future work.
2. Related Work
To gain high-quality knowledge graphs, approaches which utilize knowledge graph embedding to predict missing facts have been proposed recently. These methods fall into two broad categories in [
10]: transformation based models and semantic matching models. Specifically, transformation based models transform head entity to tail entity by relations, while semantic matching models match entities and relations semantics in latent spaces. Compared to transformation based models, semantic matching models suffer from poor interpretability.
Transformation based models usually embed entities and relations into vector space and model the relation as a transformation from head entity embeddings to tail entity embeddings. One of the most representative is TransE which mapped entities and relations to the same space
. For each triple
, entity embeddings
and relation embedding
hold
. Then a series of extensions based on TransE are presented to improve accuracy and interpretability. For instance, TransR [
11] introduced relations-specific spaces. TransR modeled relations and entities into different spaces following the idea that TransE can only express 1-to-1 relations. RotatE mapped embeddings into complex space which focused on expressing relation patterns. HAKE [
7] utilized the polar coordinate system to capture semantic hierarchies in the knowledge graphs.
Semantic matching models that match latent semantics of entities and relations can be divided into two categories: bilinear models and neural network based models. Bilinear models include DistMult [
8], HolE [
12], SimplE [
9], ComplEx [
13] and QuatE and DihEdral [
14]. DistMult represented each entity as a vector and each relation as a diagonal matrix. HolE matched latent semantics of entities by circular correlation operation and then the compositional vector interacted with relations latter. ComplEx, mapping knowledge graph embedding into complex space, leveraged Hermitian product to capture latent semantics of entities and relations which could express antisymmetry relation pattern. QuatE, extending knowledge graph embedding from complex space to quaternion space, modeled each relation as rotation in four-dimensional space with more degree of freedom. Compared with ComplEx, QuatE could express the main relation patterns except composition. For each entity, SimplE proposed two embeddings and each of them learned latent semantics dependently. DihEdral mapped relations into dihedral group to capture composition relations. Neural network based models including ConvE [
15], R-GCNs [
16] and InteractE [
17] are proposed recently. ConvE, R-GCNs introduced convolutional network and graph convolutional networks to knowledge graph embedding respectively. Compared with ConvE, InteractE introduced the feature permutation, “checkered” feature reshaping and circular convolution to increase interaction.
Recently, some models introduced hyperbolic space to knowledge graph embeddings. MuRP [
18] represented knowledge graph in Poincaré ball of hyperbolic space. Chami et al. [
19] attempted to capture hierarchical and logical patterns in hyperbolic space. Compared to hyperbolic space based models which focused on semantic hierarchies in knowledge graphs,
DualQuatE try to overcome overcome the challenge of multiple relations between two entities and relation patterns.
Both DualQuatE and QuatE use quaternion to embed knowledge graphs. However, those are three different models. The main differences between DualQuatE and QuatE are as follows:
DualQuatE, a transformation based model, measures score of triples by the distance between two entities. QuatE which is a semantic matching model measured the latent matching semantics of entities and relations.
The purpose of the model is different. DualQuatE aims to address the challenge of having multiple relations between two entities. QuatE aims to utilize quaterion Hamilton product to encourage a more compact interaction between entities and relations.
The geometric meaning is different. QuatE embeds entities and relations with quaterions to model relations as rotations. Our model firstly attempts to represent entities with pure quaternions and models relations as interaction of translation and rotation.
4. Our DualQuatE Model
In this section, we introduce our model DualQuatE which maps entities and relations to dual quaternion space, and two variations of DualQuatE, namely DualQuatE-1 and DualQuatE-2.
We denote a knowledge graph by , a set of entities by and a set of relations by . A knowledge graph is composed of a set of facts, each of which can be represented by , where is a head entity, is a tail entity, and is a relation between h and t. We denote a set of facts that are true by , and a set of facts that are false by . Given a knowledge graph , we aim to predict missing facts (i.e., link prediction) in .
4.1. Multiple Relations between the Entities
To address the challenge of having multiple relations between head and tail entities, we embed knowledge graph into dual quaternion space.
denote vector of entity embeddings and relation embeddings, each element of entity embeddings
or
is pure quaternion and every dimension of relation embeddings
is unit dual quaternion. We expect to model relation embeddings
as
interaction of
rotation and
translation from head entity embeddings
to tail entity embeddings
as shown in
Figure 2a. Specifically, each true triple
satisfies:
where each dimension of
is a unit dual quaterinon satisfying Formula (
4). We define a quaternion
to represent a rotation about pure unit quaternion
through
and a pure quaternion
. Furthermore, we define a unit dual quaternion by:
With Formula (
10), we can deduce the transformation of
DualQuatE in Formula (
9):
where the geometric meaning of
is shown in Formula (
2). As shown above,
DualQuatE transforms head entity
h to tail entity
t by relation
r which combines
rotation (i.e.,
) and
translation (i.e.,
). Unlike previous models learned similar representations of relations
shown in
Figure 2c,d, our model learns combinations of different translations and rotations to represent various relations between head and tail entities.
We define score function by:
where
represent
norm of a vector. With the score function we want head entity to be as close to tail entity as possible after the transformation of the relation.
4.2. Loss Function
We employ self-adversarial negative sampling [
5] method to generate corrupt samples. We define the probability distribution of negative samples by:
where
is sampling temperature. Combining with self-adversarial negative sampling, we define loss function by:
where
is fixed margin. We define our algorithm as shown in Algorithm 1.
Algorithm 1DualQuatE. |
Input: Entity embeddings and relation embeddings . hyperparameters including margin , martrix dim k, negative sample size n. 1: uniform for each uniform for each 2: repeat 3: uniform random sampling 4: generate n negative samples for 5: 6: compute each weight: 7: update relation embeddings and entity embeddings : 8: until
|
4.3. Properties of DualQuatE
In this part we describe the relation patterns and introduce how
DualQuatE expresses those patterns. Recently, learning relation patterns including
symmetry/antisymmetry,
inversion and
composition have been realized to the key of link prediction task. Our model
DualQuatE can easily explain the relation patterns of the learned relation embeddings and proof of relation patterns can be found in the
Appendix A.
Inversion: If a relation is the inverse to a relation , then we can infer . For example, the relation is inverse to the relation . To r and , we infer that , which denotes the composition of component and have no rotation (i.e., ) and the translation which is rotated by is the opposite number of the translation (i.e., ).
Symmetry: A relation is symmetric, if holds. For instance, relations and from the dataset WN18 are symmetric. If a relation is symmetric, we reason that , which means no rotation of the self-composition of component (i.e., ) and no translation of component (i.e., ).
Antisymmetry: A relation is antisymmetric, if , which satisfies . For example, the relation .
Composition: A relation is composed by the relation and , which can be denoted by if . For example, relation uncle_of can be composited by brother_of and father_of such as if , are true triples, we can reason is a true fact in the real world. Relation can be composited by relation and ; they can be represented by , which deduces that is equal to the sum of translation and translation which is rotated by the rotation (i.e., ).
4.4. Variations
We introduce extensions of DualQuatE. DualQuatE is a transformation based model, which combines rotation and translation. To compare the effects of interaction of rotation and translation, we compare DualQuatE with DualQuatE-1 which models relations as rotation in three-dimensional space. Furthermore, we propose DualQuatE-2 to explore the role of scaling in the rotation.
DualQuatE-1: We devise DualQuatE-1 which embeds entities and relations to quaternion space. Specifically, we represent entity embeddings with pure quaternions and relation embeddings with quaternions. We design a score function as follows to model the relation as rotation in three-dimensional space. Namely, for each fact satisfies: .
DualQuatE-2: To explore the effect of scaling in knowledge graph embeddings, we present DualQuatE-2 to introduce scaling. DualQuatE-2 maps knowledge graph embeddings to four-dimensional space. Especially, we represent entities and relations with quaternions where relation embeddings are not unit quaternions. We define score function meaning relation transform head entity to tail entity combining rotation and scaling.
4.5. Connection to TransE and RotatE
Compared with RotatE: RotatE embedded entity embeddings
,
and relation embeddings
into the complex space. RotatE utilized score function
to calculate the probability of each triple, where
is unit complex
.
DualQuatE can be transformed to RotatE by fixing rotation plane and removing translation variables. For instance, we can construct relation embeddings by Formula (
10) in
plane, where
and
(i.e.,
) and embed entities with corresponding forms:
or
.
Compared with TransE: TransE modeled relation as translation that embedded entity embeddings , and relation embeddings to vector space. To express TransE, we can set (i.e., ) in relation embeddings to ignore the rotation. In other words, the relation embedddings in DualQuatE can be expressed as .