Modeling Noncommutative Composition of Relations for Knowledge Graph Embedding

Abstract

Knowledge Graph Embedding (KGE) is a powerful way to express Knowledge Graphs (KGs), which can help machines learn patterns hidden in the KGs. Relation patterns are useful hidden patterns, and they usually assist machines to predict unseen facts. Many existing KGE approaches can model some common relation patterns like symmetry/antisymmetry, inversion, and commutative composition patterns. However, most of them are weak in modeling noncommutative composition patterns. It means these approaches can not distinguish a lot of composite relations like “father’s mother” and “mother’s father”. In this work, we propose a new KGE method called QuatRotatScalE (QRSE) to overcome this weakness, since it utilizes rotation and scaling transformations of quaternions to design the relation embedding. Specifically, we embed the relations and entities into a quaternion vector space under the difference norm KGE framework. Since the multiplication of quaternions does not satisfy the commutative law, QRSE can model noncommutative composition patterns naturally. The experimental results on the synthetic dataset also support that QRSE has this ability. In addition, the experimental results on real-world datasets show that QRSE reaches state-of-the-art in link prediction problem.

1. Introduction

Knowledge Graph (KG) is composed by structured, objective facts. The facts are usually expressed in the form of triples as (h, r, t), where h, r, and t express the head entity, the relation, and the tail entity, respectively. For example, (China, located_in, Asia). Knowledge graphs have successfully supported many applications in various fields, such as recommender systems [1], question answering [2], information retrieval [3], and natural language processing [4]. KGs have also attracted increased attention from both industry and academic communities. However, real-world knowledge graphs, such as Dbpedia [5], Freebase [6], Yago [7], and WordNet [8], are usually incomplete, which restricts their applications. Thus, knowledge graph completion has become a widely studied subject. This subject is usually formulated as a link prediction problem, i.e., predicting the missing links that should be in the knowledge graph. Generally speaking, it asks us to design an agent that takes the query as input and outputs some entities. The query may contain a head entity and a relation or a tail entity and a relation. Every outputted entity should be able to form a plausible triple together with the query.

So far, the fundamental way to deal with the link prediction problem is Knowledge Graph Embedding (KGE) in industry and academia [9,10,11,12,13]. In this way, the agent needs to learn a low-dimensional vector representation, also called embedding, for each entity and relation. We have to design a scorer that can grade any triple in the embedding form for its plausibility. When predicting the unknown entity, the agent only needs to grade all possible triples (composed by the query with each candidate entity) and then take the candidate entities of high score triples as the predicted result.

There are two reasons why these knowledge-graph embedding methods can tackle the link prediction problem effectively. On the one hand, there are many utilizable relation patterns in real-world KGs, such as symmetry/antisymmetry, inversion, and composition. These relation patterns are generally presented as natural redundancies in KGs, such as triple (China, located_in, Asia) and triple (Asia, includes, China) may exist in some KGs simultaneously, and they are describing the same fact. Here, located_in and includes are inverse relations for each other. On the other hand, existing KGE models have been able to model most relation patterns, i.e., evaluate the plausibility of triples by utilizing the relation patterns. For example, TransE [10] can model inversion patterns. When there are many natural redundancies relevant to located_in and includes in training KG, even if it has only seen triple (China, located_in, Asia) but not seen triple (Asia, includes, China), the TransE model can still mark a high plausibility score for triple (Asia, includes, China).

However, as far as we know, almost none of the existing KGE models can perfectly model the aforementioned relation patterns. For example, RotatE [13] declares that it can model all of the relation patterns, but it still has a fatal defect in modeling the composition patterns: It can only model commutative composition patterns, but can not model noncommutative composition patterns. This defect is also existing in some other KGE models which claim themselves can model composition patterns, such as TransE. Briefly speaking, a composition pattern implies the relation pattern that in the shape of $r_1\oplus r_2=r_3$, where ⊕ means the ordered composition of $r_1$ and $r_2$. If composition pattern $r_1\oplus r_2=r_3$ exists in some KG, it means that KG has frequent natural redundancies in the form of $[(e_1,r_1,e_2),(e_2,r_2,e_3),(e_1,r_3,e_3)]$, $e_i(i=1,2,3)$ can be any entity. If $r_1\oplus r_2=r_3$ and $r_2\oplus r_1=r_3$ are both in a KG, we call $r_1\oplus r_2=r_3$ a commutative composition pattern, such as $\mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{east}\_\mathtt{of}\oplus \mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{north}\_\mathtt{of}=\mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{northeast}\_\mathtt{of}$ and $\mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{north}\_\mathtt{of}\oplus \mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{east}\_\mathtt{of}=\mathtt{is}\_\mathtt{on}\_\mathtt{the}\_\mathtt{northeast}\_\mathtt{of}$. Otherwise, if only $r_1\oplus r_2=r_3$ is in the KG, it is a noncommutative composition pattern, such as $\mathtt{is}\_\mathtt{the}\_\mathtt{husband}\_\mathtt{of}\oplus \mathtt{is}\_\mathtt{the}\_\mathtt{mother}\_\mathtt{of}=\mathtt{is}\_\mathtt{the}\_\mathtt{father}\_\mathtt{of}$.

Both of RotatE and TransE model the noncommutative composition pattern as the commutative composition pattern by mistake. This mistake will bring severely ridiculous inferences. For example, they will infer (i.e., mark a high score for) triple (Mary, is_the_father_of, Barbara) based on existing triples (Mary, is_the_mother_of, James) and (James, is_the_husband_of, Barbara). The primary cause of this mistake is that they have not taken the design inspiration of their models carefully. They both expect to express a fact triple (h, r, t) through an equation relevant to the embeddings of h, r, and t (noted as boldface letters h, r, and t): $\mathbf{h}\odot \mathbf{r}=\mathbf{t}$, where ⊙ is some binary operation. Thus, they design the score function in the form of $-\parallel \mathbf{h}\odot \mathbf{r}-\mathbf{t}\parallel$, and we can see that the closer the equation is to hold, the higher the plausibility score is. TransE embeds entities and relations into the real number vector space and takes the addition in that space as ⊙, while RotatE replaces the real numbers with the complex numbers and takes the element-wise multiplication as ⊙. Since these two operations both satisfy the commutative law, the corresponding two models can only model commutative composition patterns.

Inspired by QuatE [14], which will be discussed in Section 2.1.2, we propose a new KGE model called QuatRotatScalE (or QRSE, for short) in this paper. The main difference from TransE and RotatE is it embeds entities and relations into the quaternion [15] vector space and takes the element-wise multiplication in that space as ⊙. Because the quaternion multiplication generally does not satisfy the commutative law (but there are special cases where the law holds), QRSE can model both composition patterns. Furthermore, we can prove that QRSE can also model the rest relation patterns. Thus it has become one of the KGE models that can model most relation patterns up to now. We evaluated QRSE and compared it with many baselines in two well-established and widely used real-world datasets FB15k-237 [16] and WN18RR [17]. The results indicate our method has reached the state-of-the-art in link prediction problem.

2. Related Work

At present, there are two classes of methods to solve the knowledge graph completion or link prediction problem. One class of methods is the KGE methods, and the other is the path-finding methods. Both of them are introduced below:

2.1. KGE Method

Embedding methods are widely used in many fields of machine learning since the embeddings of sentences, graphs, and many other data types can be easily transferred to various downstream tasks with only a little task-specific fine-tuning. For example, studies [18,19] first learn the embeddings of sentences and then use these embeddings to perform sentiment classification. Study [20] first learns an embedding for each graph, then use these embeddings to predict the missing labels of graphs. In addition, some other studies learn an embedding vector for each object (i.e., the node in a graph) of a given Heterogeneous Information Network (HIN) in a (semi-)supervised [21] or self-supervised [22] manner. Taking advantage of the learned embeddings of objects, they can fulfill many tasks, e.g., object classification, clustering, and visualization.

In knowledge graph completion or link prediction problem, Knowledge Graph Embedding methods are also the most studied methods. Let us use $\mathcal{E}$ to represent the set of all entities and use $\mathcal{R}$ to represent the set of all relations in KG. KGE methods need to assign a vector representation to every entity $e\in \mathcal{E}$ and relation $r\in \mathcal{R}$, noted in boldface letters $\mathbf{e}$ and $\mathbf{r}$, respectively. $\mathbf{e}$ or $\mathbf{r}$ is also called the embedding of e or r. In addition to this, KGE methods still need to design a score function $f_r(\mathbf{h},\mathbf{t})$ to mark the plausibility of the triple (h, r, t). The objective of optimization is to mark high scores for the true triples and low scores for the false triples. Based on the type of score function, we can further divide the KGE methods into two sorts, KGE based on difference norm and KGE based on semantic matching:

2.1.1. KGE Based on Difference Norm

The common motivation of this sort of method is they want to use a triple approximate equation $f_1(\mathbf{h},\mathbf{r})\approx f_2(\mathbf{t},\mathbf{r})$ to describe any triple (h, r, t), and the strict equation should hold for fact triples. As for the unknown triples, they think the proximity of the two sides can reflect the plausibility of the triple. Thus, the score functions of these methods are always in the form of $f_r(\mathbf{h},\mathbf{t})=-\parallel f_1(\mathbf{h},\mathbf{r})-f_2(\mathbf{t},\mathbf{r})\parallel$.

Among them, there is a kind of method that is widely studied, called translational methods. We call them “translational” because the origin of this kind of method, TransE, uses the translation transformation to design the triple approximate equation. Precisely, it chooses the real number vector space ${\mathbb{R}}^k$ as the embedding space and regards the relation embedding $\mathbf{r}$ as a translation transformation from head entity embedding $\mathbf{h}$ to tail entity embedding $\mathbf{t}$. So it designs the triple approximate equation as $\mathbf{h}+\mathbf{r}\approx \mathbf{t}$. Following TransE, many improvements have emerged. TransH [23] claims it is better to assign a hyperplane in embedding space for every relation (the hyperplane’s normal vector noted as ${\mathbf{r}}_p$), and only regards $\mathbf{r}$ as a translation from the projection of $\mathbf{h}$ to the projection of $\mathbf{t}$ on that hyperplane. Hence the triple approximate equation of TransH is $(\mathbf{I}-{\mathbf{r}}_p{\mathbf{r}}_p^{\top })\mathbf{h}+\mathbf{r}\approx (\mathbf{I}-{\mathbf{r}}_p{\mathbf{r}}_p^{\top })\mathbf{t}$, where $\mathbf{I}$ is the identity matrix. TransR [24] generalizes TransH, it assigns a linear map to every relation r, noted as transfer matrix ${\mathbf{W}}_r$. This linear map maps $\mathbf{h}$ and $\mathbf{t}$ into the relation space. Then TransR utilizes the images of $\mathbf{h}$ and $\mathbf{t}$ in the relation space with $\mathbf{r}$ to design the triple approximate equation in TransE’s style: ${\mathbf{W}}_r\mathbf{h}+\mathbf{r}\approx {\mathbf{W}}_r\mathbf{t}$. Further, StransE [25] assigns each relation r two different transfer matrices ${\mathbf{W}}_{r,1}$ and ${\mathbf{W}}_{r,2}$. Similarly, the triple approximate equation is designed as ${\mathbf{W}}_{r,1}\mathbf{h}+\mathbf{r}\approx {\mathbf{W}}_{r,2}\mathbf{t}$. These derivative methods of TransE are collectively known as TransX. Their score functions can be written in the form of $f_r(\mathbf{h},\mathbf{t})=-\parallel g_{r,1}\left(\mathbf{h}\right)+\mathbf{r}-g_{r,2}\left(\mathbf{t}\right)\parallel$, where $g_{r,i}(·)$ denotes a matrix multiplication concerning relation r.

Since the large number of the derivative methods of TransE, some literature uses the translational methods to refer to KGE based on difference norm in general. But this is not accurate enough. Some other methods do not turn to translation transformation to design their triple approximate equations, such as TorusE [26] and RotatE. TorusE chooses a compact Lie group as its embedding space and can be regarded as a special case of RotatE when the embedding modulus are fixed [13]. RotatE embeds entities and relations into the complex number vector space ${\mathbb{C}}^k$. It wants to replace the translation in ${\mathbb{R}}^k$ with the rotation in ${\mathbb{C}}^k$. Specifically, for each element $r_i(1\le i\le k)$ of $\mathbf{r}$, RotatE fixes it as a unitary complex number (i.e., $|r_i|=1$). Hence the complex multiplication between the i-th element of $\mathbf{h}$ (i.e., $h_i$) and $r_i$ means $h_i$ rotates in its complex plane with angle $Arg\left(r_i\right)$ (i.e., the argument of complex $r_i$). Let us use ∘ to denote the Hadamard (element-wise) product between two complex vectors, the triple approximate equation of RotatE is $\mathbf{h}\circ \mathbf{r}\approx \mathbf{t}$.

There are some KGE methods with score functions belonging to a special case of difference norm, which is in the form of $-\parallel \mathbf{h}\odot \mathbf{r}-\mathbf{t}\parallel$, where ⊙ is some binary operation. When the ideal optimization is achieved, the triple approximate equations of these methods hold: $\mathbf{h}\odot \mathbf{r}=\mathbf{t}$. This property is useful to explain some abilities to model relation patterns. For example, TransE and RotatE are two of these methods, and because their binary operations are both associative and commutative, they can only model commutative composition patterns. For more details, please see Section 5.

2.1.2. KGE Based on Semantic Matching

The intuition of this sort of method is to measure the plausibility of a triple by inspecting the matching degree of the latent semantics of the two entities and the relation.

There is a family of methods called bilinear models that design score functions as bilinear maps of head and tail entities. RESCAL [9] may be the first bilinear model. It selects real vector space ${\mathbb{R}}^k$ as the embedding space of entities and assigns a $k\times k$ real matrix ${\mathbf{W}}_r$ to each relation r. Then it directly applies ${\mathbf{W}}_r$ to define a bilinear map as the score function. To reduce the complexity of ${\mathbf{W}}_r$, DistMult [11] restricts ${\mathbf{W}}_r$ to be a diagonal matrix. So DistMult can express ${\mathbf{W}}_r$ as a vector $\mathbf{r}$ in ${\mathbb{R}}^k$ and rewrite the score function in the form of the multi-linear dot product of $\mathbf{h}$, $\mathbf{r}$, and $\mathbf{t}$. To overcome DistMult’s weakness in modeling antisymmetry relation pattern, ComplEx [12] extends the embedding space into the complex vector space ${\mathbb{C}}^k$, and modifies the score function. QuatE [14] further develops ComplEx, it extends the embedding space into the quaternion vector space to obtain better expression ability. DualE [27] uses the dual quaternion vectors to design the embeddings of entities and relations, and chooses the dual quaternion inner product as the score function. DihEdral [28] designs entity embeddings with real vectors, and designs relation embeddings with dihedral group vectors, where each dihedral group is expressed as a second-order discrete real matrix. Although its score function is a bilinear form, which belongs to the type of semantic matching, it is theoretically proven that this score function is equivalent to a difference norm function in the form of $-\parallel \mathbf{h}\odot \mathbf{r}-\mathbf{t}\parallel$ for optimizing relation embeddings. So DihEdral has the ability to model composition patterns like TransE and RotatE. Furthermore, since the multiplication of dihedral groups generally does not satisfy the commutative law, DihEdral can model noncommutative composition patterns. However, because the relation embeddings take discrete values, DihEdral has to use special treatments of the relation embeddings during the training process, and the actual performance is easily affected by special treatments. As for QuatE and DualE, their relation embeddings have the potential to model noncommutative composition patterns for the (dual) quaternion multiplication generally does not satisfy the commutative law. Nevertheless, because their score functions belong to the type of semantic matching and lack the theoretical equivalence to a difference norm function in the form of $-\parallel \mathbf{h}\odot \mathbf{r}-\mathbf{t}\parallel$ like DihEdral at present, their abilities to model the composition patterns have no strict theoretical guarantees. More precisely, their triple approximate equations, if any, do not necessarily hold when the ideal optimization is achieved, which is a crucial but easily overlooked step for a rigorous proof.

Apart from bilinear models, some models based on neural networks emerged recently. Such as ConvE [17] and ConvKB [29] take the convolutional neural networks to construct the score functions.

Some mentioned KGE methods are listed in

Table 1. Score functions and embedding spaces of several KGE models. $\langle \mathbf{a},\mathbf{b},\mathbf{c}\rangle \doteq {\sum }_{i=1}^k{a}_i{b}_i{c}_i$ means the multi-linear dot product of vector $\mathbf{a}$, $\mathbf{b}$, and $\mathbf{c}$; $\overline{·}$ denotes conjugate for a complex or quaternion vectors; $\mathrm{Re}(·)$ denotes the real part of a complex number or quaternion; ⊗ indicates the Hadamard (element-wise) product between two quaternion vectors. Note that we report an equivalent formulation for QuatE to show the inheritance relationship with ComplEx.

Model	Score Function	Embedding Space
TransE	$-\parallel \mathbf{h}+\mathbf{r}-\mathbf{t}\parallel$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{R}}^k$
TransX	$-\parallel g_{r,1}\left(\mathbf{h}\right)+\mathbf{r}-g_{r,2}\left(\mathbf{t}\right)\parallel$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{R}}^k$
RotatE	$-\parallel \mathbf{h}\circ \mathbf{r}-\mathbf{t}\parallel$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{C}}^k,\left|r_i\right|=1$
RESCAL	${\mathbf{h}}^{\top }{\mathbf{W}}_r\mathbf{t}$	$\mathbf{h},\mathbf{t}\in {\mathbb{R}}^k,{\mathbf{W}}_r\in {\mathbb{R}}^{k\times k}$
DistMult	$\langle \mathbf{h},\mathbf{r},\mathbf{t}\rangle$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{R}}^k$
ComplEx	$\mathrm{Re}\left(\langle \mathbf{h},\mathbf{r},\overline{\mathbf{t}}\rangle \right)$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{C}}^k$
QuatE	$\mathrm{Re}\left(\langle \mathbf{h},\mathbf{r},\overline{\mathbf{t}}\rangle \right)$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{H}}^k,\left|r_i\right|=1$
QRSE	$-\parallel \mathbf{h}\otimes \mathbf{r}-\mathbf{t}\parallel$	$\mathbf{h},\mathbf{r},\mathbf{t}\in {\mathbb{H}}^k$

with their score functions. Their abilities to model the relation patterns are shown in

Table 2. The modeling ability comparison for various relation patterns among different models (partial reference from [13]).

Model	Symmetry	Anti- Symmetry	Inversion	Commutative Composition	Noncommutative Composition
TransE	×	√	√	√	×
TransX	√	√	×	×	×
RotatE	√	√	√	√	×
RESCAL	√	√	√	×	×
DistMult	√	×	×	×	×
ComplEx	√	√	√	×	×
QuatE	√	√	√	×	×
QRSE	√	√	√	√	√

. We can see that our QRSE can model all relation patterns, which is a rare ability.

Additionally, “supervised relation composition” [30] is a method that can model composition patterns under supervision. But it is not a KGE method. Its goal is to design and train a function model that can take the embeddings of two relations as input and output the embedding of the composite relation of these two relations. The relation embeddings used are provided by an existing KGE model and are fixed once obtained. The supervisory information used for training is mined from the original KGs by another method. This method and the KGE models mentioned before belong to different research directions. The direction of KGE models studies how to directly model relation patterns (including composition patterns) by training entity and relation embeddings from the original KGs.

2.2. Path-Finding Method

This class of methods does not need score function to predict unknown entities, such as MINERVA [31], MultiHopKG [32], and DeepPath [33]. Instead, they should start from the query entity node and follow the direction implied by the query relation to search the KG for the unknown entity. Compared with KGE methods, their results are explainable to some extent since they can provide the inference paths as evidence, but the lack of precision is their weakness at present.

3. Preliminaries

Before introducing our proposed method, let us briefly explain the related concepts and geometric meaning of quaternions.

3.1. A Brief Introduction of Quaternion

As an extened number system from the complex numbers $\mathbb{C}$, quaternions $\mathbb{H}$ [15] have to import three fundamental quaternion units $\mathit{i}$, $\mathit{j}$, and $\mathit{k}$, which are not existing in the real numbers. Each quaternion q can be expressed as $q=a+b\mathit{i}+c\mathit{j}+d\mathit{k}$, where a, b, c, and d are all real numbers. The addition of quaternions is defined as $(a_1+b_1\mathit{i}+c_1\mathit{j}+d_1\mathit{k})+(a_2+b_2\mathit{i}+c_2\mathit{j}+d_2\mathit{k})\doteq (a_1+a_2)+(b_1+b_2)\mathit{i}+(c_1+c_2)\mathit{j}+(d_1+d_2)\mathit{k}$. The multiplication between any two fundamental quaternion units are defined as ${\mathit{i}}^2={\mathit{j}}^2={\mathit{k}}^2=-1$ and $\mathit{i}\mathit{j}=-\mathit{j}\mathit{i}=\mathit{k},\mathit{j}\mathit{k}=-\mathit{k}\mathit{j}=\mathit{i},\mathit{k}\mathit{i}=-\mathit{i}\mathit{k}=\mathit{j}$. Obviously, this multiplication is associative but not commutative. For completeness, we also confirm the multiplication between any one in $\{\mathit{i},\mathit{j},\mathit{k}\}$, and a real number is commutative and associative. To obey the distributive law, we consequently get the multiplication between two arbitrary quaternions as:

$$\begin{array}{cc}\hfill & (a_1+b_1\mathit{i}+c_1\mathit{j}+d_1\mathit{k})(a_2+b_2\mathit{i}+c_2\mathit{j}+d_2\mathit{k})\hfill \\ \hfill & \doteq a_1{a}_2-b_1{b}_2-c_1{c}_2-d_1{d}_2\hfill \\ \hfill & +(a_1{b}_2+b_1{a}_2+c_1{d}_2-d_1{c}_2)\mathit{i}\hfill \\ \hfill & +(a_1{c}_2+c_1{a}_2+d_1{b}_2-b_1{d}_2)\mathit{j}\hfill \\ \hfill & +(a_1{d}_2+d_1{a}_2+b_1{c}_2-c_1{b}_2)\mathit{k}.\hfill \end{array}$$

(1)

We can conclude that the multiplication of quaternions (also known as the Hamilton product) holds the associative and distributive law, but does not hold the commutative law in general. Nevertheless, there are some special cases where the commutative law holds.

Some useful concepts of quaternions are listed as follows (let $q=a+b\mathit{i}+c\mathit{j}+d\mathit{k}$):

Modulus: The modulus of q is written as $\left|q\right|$ and is defined as $\left|q\right|\doteq \sqrt{a^2+b^2+c^2+d^2}$. Since the set of quaternions $\mathbb{H}$ is a linear space isomorphic to ${\mathbb{R}}^4$ with basis $(1,\mathit{i},\mathit{j},\mathit{k})$, modulus means the length of q intuitively. In addition, if $\left|q\right|=1$, q is called a unit quaternion.

Real and imaginary part: Similar to complex numbers, real number a is the real part of q, and real vector $\mathbf{v}\doteq {(b,c,d)}^{\top }$ is the imaginary part of q. Sometimes we would like to express q in the form of $[a,\mathbf{v}]$ for convenience. Then, the multiplication of quaternions can be written as $[a_1,{\mathbf{v}}_1][a_2,{\mathbf{v}}_2]=[a_1{a}_2-{\mathbf{v}}_1·{\mathbf{v}}_2,a_1{\mathbf{v}}_2+a_2{\mathbf{v}}_1+{\mathbf{v}}_1\times {\mathbf{v}}_2]$, where · is the dot product and × is the cross product.

Conjugate: The conjugate of q is the quaternion $\overline{q}\doteq a-b\mathit{i}-c\mathit{j}-d\mathit{k}$. It has thses properties: (1) $\overline{q_1{q}_2}=\overline{q_2}\overline{q_1}$; (2)$q\overline{q}=\overline{q}q={\left|q\right|}^2$, and from (1), (2) we get (3) $|q_1\left|\right|q_2|=|q_1{q}_2|$. As a corollary, the product of two unit quaternions is also a unit quaternion.

Reciprocal: If $q\ne 0$, the reciprocal of q is the quaternion $q^{-1}$ such that $q{q}^{-1}=q^{-1}q=1$, and it is equivalent to define $q^{-1}\doteq \overline{q}/{\left|q\right|}^2$.

3.2. The Geometric Meaning of the Multiplication of Quaternions

To see the geometric meaning of the multiplication of quaternions, we have to view the $\mathbb{H}$ as a linear space isomorphic to ${\mathbb{R}}^4$ with an orthonormal basis $(1,\mathit{i},\mathit{j},\mathit{k})$. Any q in $\mathbb{H}$ can be expressed in the form as $\rho [cos\theta ,sin\theta \mathbf{n}]$, where $\rho \ge 0$ and $\parallel \mathbf{n}\parallel =1$. This is because if $q\ne 0$ we could set $\rho =\left|q\right|$, $\theta =arccos(a/|q\left|\right)$, and $\mathbf{n}=\mathbf{v}/\parallel \mathbf{v}\parallel$, whereas if $q=0$ we could set $\rho =0$ and choose $\theta$ and $\mathbf{n}$ arbitrarily. Note that $[cos\theta ,sin\theta \mathbf{n}]$ is a unit quaternion and implies the direction of q, while $\rho$ implies the length of q.

Take another quaternion $p=[s,\mathbf{u}]$ from $\mathbb{H}$, then the product $pq=\rho \left(p\right[cos\theta ,sin\theta \mathbf{n}\left]\right)$ means a new quaternion reached via two steps from p: (1) changing the direction of p according to $[cos\theta ,sin\theta \mathbf{n}]$, (2) stretching the length by $\rho$ times. So we only have to see what is the change implied by $p[cos\theta ,sin\theta \mathbf{n}]$.

Without loss of generality, we suppose $\mathbf{u}\ne \mathbf{0}$ and $\mathbf{u}$ is not parallel with $\mathbf{n}$. Thus we can find another orthonormal basis of $\mathbb{H}$: $([1,\mathbf{0}],[0,\mathbf{n}],[0,{\mathbf{n}}_{\perp }],[0,{\mathbf{n}}_{\times }])$. Here, ${\mathbf{n}}_{\perp }\doteq (\mathbf{u}-(\mathbf{u}·\mathbf{n})\mathbf{n})/\parallel \mathbf{u}-(\mathbf{u}·\mathbf{n})\mathbf{n}\parallel$ and ${\mathbf{n}}_{\times }\doteq {\mathbf{n}}_{\perp }\times \mathbf{n}$. Besides, we confirm the coordinates of p under this basis is ${(s,l,l_{\perp },0)}^{\top }$, where $l=\mathbf{u}·\mathbf{n}$ and $l_{\perp }=\parallel \mathbf{u}-(\mathbf{u}·\mathbf{n})\mathbf{n}\parallel$. Thus we can split p into two parts: $p=p_1+p_2$, where $p_1=[s,l\mathbf{n}]$ and $p_2=[0,l_{\perp }{\mathbf{n}}_{\perp }]$. So we only have to see what do $p_1[cos\theta ,sin\theta \mathbf{n}]$ and $p_2[cos\theta ,sin\theta \mathbf{n}]$ mean.

Since $p_1[cos\theta ,sin\theta \mathbf{n}]=[scos\theta -lsin\theta ,(ssin\theta +lcos\theta )\mathbf{n}]$, this product and $p_1$ are both in the plane with basis $\left(\right[1,\mathbf{0}],[0,\mathbf{n}\left]\right)$. And we can show the transformation from $p_1$ to the product by their coordinates under basis $\left(\right[1,\mathbf{0}],[0,\mathbf{n}\left]\right)$ as:

$$p_1\to p_1[cos\theta ,sin\theta \mathbf{n}]:\left(\begin{array}{c}s\\ l\end{array}\right)⟶\left(\begin{array}{c}scos\theta -lsin\theta \\ ssin\theta +lcos\theta \end{array}\right)=\left(\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right)\left(\begin{array}{c}s\\ l\end{array}\right).$$

(2)

So $p_1[cos\theta ,sin\theta \mathbf{n}]$ means $p_1$ rotates with angle $\theta$ counterclockwise in plane $span\left(\right[1,\mathbf{0}],[0,\mathbf{n}\left]\right)$. In the same way, since $p_2[cos\theta ,sin\theta \mathbf{n}]=[0,l_{\perp }cos\theta {\mathbf{n}}_{\perp }+l_{\perp }sin\theta {\mathbf{n}}_{\times }]$, this product and $p_2$ are both in the plane with basis $([0,{\mathbf{n}}_{\perp }],[0,{\mathbf{n}}_{\times }])$. And we can show the transformation from $p_2$ to the product by their coordinates under basis $([0,{\mathbf{n}}_{\perp }],[0,{\mathbf{n}}_{\times }])$ as:

$$p_2\to p_2[cos\theta ,sin\theta \mathbf{n}]:\left(\begin{array}{c}{l}_{\perp }\\ 0\end{array}\right)⟶\left(\begin{array}{c}{l}_{\perp }cos\theta \\ l_{\perp }sin\theta \end{array}\right)=\left(\begin{array}{cc}cos\theta & -sin\theta \\ sin\theta & cos\theta \end{array}\right)\left(\begin{array}{c}{l}_{\perp }\\ 0\end{array}\right).$$

(3)

So $p_2[cos\theta ,sin\theta \mathbf{n}]$ means $p_2$ rotates with angle $\theta$ counterclockwise in plane $span([0,{\mathbf{n}}_{\perp }],[0,{\mathbf{n}}_{\times }])$.

In a word, the change implied by $p[cos\theta ,sin\theta \mathbf{n}]$ is: (1) Split p into two components $p_1$ and $p_2$, where $p_1$ is in plane $span\left(\right[1,\mathbf{0}],[0,\mathbf{n}\left]\right)$ and $p_2$ is in plane $span([0,{\mathbf{n}}_{\perp }],[0,{\mathbf{n}}_{\times }])$; (2) Rotate $p_1$ and $p_2$ with angle $\theta$ counterclockwise in each plane simultaneously, as shown in Figure 1; (3) Add two new components together.

As a special case, when $\mathbf{u}=\mathbf{0}$ or $\mathbf{u}$ is parallel with $\mathbf{n}$, $p=p_1$. Thus at that time, $p[cos\theta ,sin\theta \mathbf{n}]$ only means the rotation in plane $span\left(\right[1,\mathbf{0}],[0,\mathbf{n}\left]\right)$. Moreover, the geometric meaning of $qp$ is almost the same with $pq$, except that the rotation for $p_2$ is clockwise.

4. Proposed Method

Now we start to introduce our proposed KGE model. The embedding spaces of entities $\mathcal{E}$ and relations $\mathcal{R}$ are both the quaternion vector space ${\mathbb{H}}^k$. For any $e\in \mathcal{E}$ and $r\in \mathcal{R}$, their embeddings are noted as $\mathbf{e}$ and $\mathbf{r}$ in lower case bold letters, respectively. The i-th elements of $\mathbf{e}$ and $\mathbf{r}$ are written as $e_i$ ($e_i\in \mathbb{H}$) and $r_i$ ($r_i\in \mathbb{H}$) for every integer i from 1 to k. Our model is based on difference norm, so we first define its triple approximate equation as $\mathbf{h}\otimes \mathbf{r}\approx \mathbf{t}$ for any triple (h, r, t). Here, ⊗ denotes the Hadamard (element-wise) product between two quaternion vectors. So this triple approximate equation is equivalent to asking for $h_i{r}_i\approx t_i$ for all $i(1\le i\le k)$. In consequence, we get our score function:

$$f_r(\mathbf{h},\mathbf{t})\doteq -\parallel \mathbf{h}\otimes \mathbf{r}-\mathbf{t}\parallel .$$

(4)

Here, $\parallel \mathbf{q}\parallel$ is the abbreviation of ${\parallel \mathbf{q}\parallel }_{p,1}\doteq {\sum }_{i=1}^k{({|a_i|}^p+{|b_i|}^p+{|c_i|}^p+{|d_i|}^p)}^{\frac{1}{p}}$, for any quaternion vector $\mathbf{q}(q_i=a_i+b_i\mathit{i}+c_i\mathit{j}+d_i\mathit{k},1\le i\le k)$. $p(p\ge 1)$ is a hyperparameter.

According to the geometric meaning of the quaternion multiplication, we can explain the purpose of this triple approximate equation intuitively: We treat each element of relation embedding $r_i$ (written in the form of ${\rho }_i[cos{\theta }_i,sin{\theta }_i{\mathbf{n}}_i]$) as a two-step transformation from $h_i$ to $t_i$: (1) Rotate $h_i$ in two planes ($span([1,\mathbf{0}],[0,{\mathbf{n}}_i])$ and $span([0,{\mathbf{n}}_{i,\perp }],[0,{\mathbf{n}}_{i,\times }])$) counterclockwise with angle ${\theta }_i$; (2) Stretch $h_i$ with scaling factor ${\rho }_i$. Thus we refer to our model as QuatRotatScalE (or QRSE, for short) due to we use Quaternions with Rotation and Scaling transformations to design the Embedding model.

Optimization

The general objective of KGE models is to return high scores for true triples and low scores for false triples. We adopt negative sampling as our training style to avoid the efficiency loss brought by the huge number of entities like most of the other KGE methods. The training KG usually only contains true triples (positive samples, noted as $\Omega$) without false triples (negative samples). Thus we apply a common way (i.e., corrupting the positive samples) to obtain the negative samples. Suppose (h, r, t) is a positive sample, we can get two sets of negative samples by replacing the head or tail entity with other entities: ${\mathcal{N}}_h(r,t)\doteq \{(h^{\prime },r,t)\mid h^{\prime }\mathrm{is}\mathrm{uniformly}\mathrm{sampled}\mathrm{from}\mathcal{E}\mathrm{s}.\mathrm{t}.(h^{\prime },r,t)\notin \Omega \}$ and ${\mathcal{N}}_t(h,r)\doteq \{(h,r,t^{\prime })\mid t^{\prime }\mathrm{is}\mathrm{uniformly}\mathrm{sampled}\mathrm{from}\mathcal{E}\mathrm{s}.\mathrm{t}.(h,r,t^{\prime })\notin \Omega \}$. The size of negative samples ${\mathcal{N}}_h(r,t)$ and ${\mathcal{N}}_t(h,r)$ is fixed and much smaller than $\left|\mathcal{E}\right|$.

Following RotatE [13], we use the loss function on each triple (h, r, t) in the training KG as

$$L=-log\sigma (\gamma +f_r(\mathbf{h},\mathbf{t}))-\sum _{(h^{\prime },r,t^{\prime })\in \mathcal{N}}p(h^{\prime },r,t^{\prime })log\sigma (-\gamma -f_r({\mathbf{h}}^{\prime },{\mathbf{t}}^{\prime }))$$

(5)

where $\sigma$ is the sigmoid function, $\gamma$ is a fixed margin, and $\mathcal{N}$ is ${\mathcal{N}}_h(r,t)$ or ${\mathcal{N}}_t(h,r)$. In practice, $\mathcal{N}$ is regenerated in the same way (${\mathcal{N}}_h(r,t)$ or ${\mathcal{N}}_t(h,r)$) for every positive sample in one training batch. Once it turns to the next training batch, $\mathcal{N}$ should switch the regenerating way. $p(h^{\prime },r,t^{\prime })$ is the distribution of self-adversarial negative sampling proposed by RotatE [13] and is defined as

$$p(h^{\prime },r,t^{\prime })=\frac{exp\left(\alpha f_r({\mathbf{h}}^{\prime },{\mathbf{t}}^{\prime })\right)}{{\sum }_{(h^{\prime },r,t^{\prime })\in \mathcal{N}}exp\left(\alpha f_r({\mathbf{h}}^{\prime },{\mathbf{t}}^{\prime })\right)}$$

(6)

where $\alpha$ is the temperature of sampling. The self-adversarial negative sampling can moderate the low efficiency of the uniform negative sampling. We also take Adam as our optimizer. Moreover, $p(h^{\prime },r,t^{\prime })$ plays a role of importance sampling ratio in L, so it need not backpropagate gradients through it.

5. Theoretic Analysis

5.1. Relation Patterns

As mentioned in introduction Section 1, modeling (i.e., identifying and utilizing) the relation patterns in KGs are the fundamental for KGE models to solve the link prediction problem. There are three types of relation patterns, which are very powerful and widely exist in various KGs [12,13,16,34,35]:

Symmetry/antisymmetry: A relation r is symmetric (antisymmetric) if $\forall e_1,e_2\in \mathcal{E}$, $(e_1,r,e_2)\Rightarrow (e_2,r,e_1)$ ($(e_1,r,e_2)\Rightarrow \neg (e_2,r,e_1)$).

Inversion: Relation $r_1$ is inverse to relation $r_2$ if $\forall e_1,e_2\in \mathcal{E}$, $(e_1,r_2,e_2)\Rightarrow (e_2,r_1,e_1)$.

Composition: Relation $r_1$ is composed of relation $r_2$ and relation $r_3$ if $\forall e_1,e_2,e_3\in \mathcal{E}$, $(e_1,r_2,e_2)\wedge (e_2,r_3,e_3)\Rightarrow (e_1,r_1,e_3)$. We adopt form $r_2\oplus r_3=r_1$ to describe this composition pattern for simplicity. Moreover, if both of $r_2\oplus r_3=r_1$ and $r_3\oplus r_2=r_1$ make sense, $r_2\oplus r_3=r_1$ is a commutative composition pattern. Otherwise, if only $r_2\oplus r_3=r_1$ holds, it is a noncommutative composition pattern.

5.2. Abilities to Model Relation Patterns

In this subsection, we prove that QRSE can model symmetry/antisymmetry, inversion, and composition patterns. Additionally, TransE and RotatE are unable to model noncommutative composition patterns. Next, if triple (h, r, t) is in the knowledge graph, we write it in the embedding space as $\mathbf{h}\otimes \mathbf{r}=\mathbf{t}$ for QRSE because its score function is a special case of difference norm $-\parallel \mathbf{h}\otimes \mathbf{r}-\mathbf{t}\parallel$ and when the ideal optimization is achieved, we can directly get $\mathbf{h}\otimes \mathbf{r}=\mathbf{t}$ (we can replace ⊗ with + or ∘ for TransE or RotatE for the same reason).

• QRSE can model symmetry/antisymmetry patterns:
  Suppose ${\mathbf{e}}_2\otimes \mathbf{r}={\mathbf{e}}_1$ and ${\mathbf{e}}_1\otimes \mathbf{r}={\mathbf{e}}_2$. We can get ${\mathbf{e}}_1\otimes \mathbf{r}\otimes \mathbf{r}={\mathbf{e}}_1$. It means for any $i(1\le i\le k)$, $e_{1,i}{r}_i{r}_i=e_{1,i}$. If $e_{1,i}=0$, $r_i$ can be any quaternion. But if $e_{1,i}\ne 0$, $r_i$ must satisfies:

  $$e_{1,i}{r}_i{r}_i=e_{1,i}\iff e_{1,i}^{-1}{e}_{1,i}{r}_i{r}_i=e_{1,i}^{-1}{e}_{1,i}\iff r_i{r}_i=1\iff r_i=r_i^{-1}.$$

  (7)
  $\because \forall q_1,q_2\in \mathbb{H},|q_1\left|\right|q_2|=|q_1{q}_2|$, $\therefore |r_i|=1$. $\because \forall q\in \mathbb{H},q^{-1}=\overline{q}/{\left|q\right|}^2$, $\therefore r_i=\overline{r_i}$. Thus $r_i$ is 1 or $-1$. In a word, if $\mathbf{r}$ satisfies $r_i\in \{1,-1\}(1\le i\le k)$, $\mathbf{r}$ models a symmetry pattern, otherwise, it models a antisymmetry pattern.
• QRSE can model inversion patterns:
  Suppose ${\mathbf{e}}_2\otimes {\mathbf{r}}_1={\mathbf{e}}_1$ and ${\mathbf{e}}_1\otimes {\mathbf{r}}_2={\mathbf{e}}_2$. We can get ${\mathbf{e}}_1\otimes {\mathbf{r}}_2\otimes {\mathbf{r}}_1={\mathbf{e}}_1$. It means for any $i(1\le i\le k)$, $e_{1,i}{r}_{2,i}{r}_{1,i}=e_{1,i}$. If $e_{1,i}=0$, $r_{2,i}$ and $r_{1,i}$ can be any quaternions. But if $e_{1,i}\ne 0$, $r_{2,i}$ and $r_{1,i}$ must satisfy:

  $$\begin{array}{cc}\hfill & e_{1,i}{r}_{2,i}{r}_{1,i}=e_{1,i}\iff e_{1,i}^{-1}{e}_{1,i}{r}_{2,i}{r}_{1,i}=e_{1,i}^{-1}{e}_{1,i}\hfill \\ \hfill & \iff r_{2,i}{r}_{1,i}=1\iff r_{1,i}=r_{2,i}^{-1}.\hfill \end{array}$$

  (8)
  Define ${\mathbf{q}}^{-1}\doteq {(q_1^{-1},q_2^{-1},...,q_k^{-1})}^{\top }$ for all $\mathbf{q}\in {\mathbb{H}}^k,q_i\ne 0(1\le i\le k)$. We can conclude that if ${\mathbf{r}}_1={\mathbf{r}}_2^{-1}$, ${\mathbf{r}}_1$ and ${\mathbf{r}}_2$ model an inversion pattern.
• QRSE can model composition patterns:
  Suppose ${\mathbf{e}}_1\otimes {\mathbf{r}}_2={\mathbf{e}}_2$, ${\mathbf{e}}_2\otimes {\mathbf{r}}_3={\mathbf{e}}_3$, and ${\mathbf{e}}_1\otimes {\mathbf{r}}_1={\mathbf{e}}_3$. We can get ${\mathbf{e}}_1\otimes {\mathbf{r}}_2\otimes {\mathbf{r}}_3={\mathbf{e}}_1\otimes {\mathbf{r}}_1$. It means for any $i(1\le i\le k)$, $e_{1,i}{r}_{2,i}{r}_{3,i}=e_{1,i}{r}_{1,i}$. If $e_{1,i}=0$, $r_{2,i}$, $r_{3,i}$, and $r_{1,i}$ can be any quaternions. But if $e_{1,i}\ne 0$, $r_{2,i}$, $r_{3,i}$, and $r_{1,i}$ must satisfy:

  $$e_{1,i}{r}_{2,i}{r}_{3,i}=e_{1,i}{r}_{1,i}\iff e_{1,i}^{-1}{e}_{1,i}{r}_{2,i}{r}_{3,i}=e_{1,i}^{-1}{e}_{1,i}{r}_{1,i}\iff r_{2,i}{r}_{3,i}=r_{1,i}.$$

  (9)
  Moreover, if we still suppose ${\mathbf{e}}_4\otimes {\mathbf{r}}_3={\mathbf{e}}_5$, ${\mathbf{e}}_5\otimes {\mathbf{r}}_2={\mathbf{e}}_6$, and ${\mathbf{e}}_4\otimes {\mathbf{r}}_1={\mathbf{e}}_6$. Then if $e_{4,i}\ne 0$ for all $i(1\le i\le k)$, $r_{3,i}$, $r_{2,i}$, and $r_{1,i}$ must satisfy: $r_{3,i}{r}_{2,i}=r_{1,i}$. This means $r_{2,i}{r}_{3,i}=r_{3,i}{r}_{2,i}$. If we note $r_{2,i}=[a_{2,i},{\mathbf{v}}_{2,i}]$ and $r_{3,i}=[a_{3,i},{\mathbf{v}}_{3,i}]$, then we will get:

  $$\begin{array}{cc}\hfill & [a_{2,i}{a}_{3,i}-{\mathbf{v}}_{2,i}·{\mathbf{v}}_{3,i},a_{2,i}{\mathbf{v}}_{3,i}+a_{3,i}{\mathbf{v}}_{2,i}+{\mathbf{v}}_{2,i}\times {\mathbf{v}}_{3,i}]\hfill \\ \hfill =& [a_{3,i}{a}_{2,i}-{\mathbf{v}}_{3,i}·{\mathbf{v}}_{2,i},a_{3,i}{\mathbf{v}}_{2,i}+a_{2,i}{\mathbf{v}}_{3,i}+{\mathbf{v}}_{3,i}\times {\mathbf{v}}_{2,i}]\hfill \\ \hfill & \iff {\mathbf{v}}_{2,i}\times {\mathbf{v}}_{3,i}={\mathbf{v}}_{3,i}\times {\mathbf{v}}_{2,i}\hfill \\ \hfill & \iff {\mathbf{v}}_{2,i}=0\mathrm{or}{\mathbf{v}}_{3,i}=\lambda {\mathbf{v}}_{2,i}(\lambda \in \mathbb{R}).\hfill \end{array}$$

  (10)
  We can conclude that if ${\mathbf{r}}_2\otimes {\mathbf{r}}_3={\mathbf{r}}_1$, ${\mathbf{r}}_2$, ${\mathbf{r}}_3$, and ${\mathbf{r}}_1$ model a composition pattern. Moreover, if ${\mathbf{v}}_{2,i}$ is parallel with ${\mathbf{v}}_{3,i}$ for all $i(1\le i\le k)$, it is a commutative composition pattern, otherwise, it is a noncommutative composition pattern.
• TransE and RotatE can not model noncommutative composition patterns, and they can only model commutative composition patterns:
  For TransE, we suppose ${\mathbf{e}}_1+{\mathbf{r}}_2={\mathbf{e}}_2$, ${\mathbf{e}}_2+{\mathbf{r}}_3={\mathbf{e}}_3$, ${\mathbf{e}}_1+{\mathbf{r}}_1={\mathbf{e}}_3$, ${\mathbf{e}}_4+{\mathbf{r}}_3={\mathbf{e}}_5$, ${\mathbf{e}}_5+{\mathbf{r}}_2={\mathbf{e}}_6$, but ${\mathbf{e}}_4+{\mathbf{r}}_1\ne {\mathbf{e}}_6$, which means the composition of relation ${\mathbf{r}}_2$ and ${\mathbf{r}}_3$ is noncommutative. From the first three equations we get ${\mathbf{r}}_2+{\mathbf{r}}_3={\mathbf{r}}_1$, and from the fourth and fifth equations we get ${\mathbf{e}}_4+{\mathbf{r}}_3+{\mathbf{r}}_2={\mathbf{e}}_6$. Because ${\mathbf{r}}_2+{\mathbf{r}}_3={\mathbf{r}}_3+{\mathbf{r}}_2$, we get ${\mathbf{e}}_4+{\mathbf{r}}_1={\mathbf{e}}_6$, which contradicts the condition. Therefore TransE can not model noncommutative composition patterns. If we replace the condition ${\mathbf{e}}_4+{\mathbf{r}}_1\ne {\mathbf{e}}_6$ with ${\mathbf{e}}_4+{\mathbf{r}}_1={\mathbf{e}}_6$, then the composition of relation ${\mathbf{r}}_2$ and ${\mathbf{r}}_3$ becomes commutative composition. In this case the previous contradiction disappears, which means TransE can model commutative composition patterns.
  As for RotatE, we suppose ${\mathbf{e}}_1\circ {\mathbf{r}}_2={\mathbf{e}}_2$, ${\mathbf{e}}_2\circ {\mathbf{r}}_3={\mathbf{e}}_3$, ${\mathbf{e}}_1\circ {\mathbf{r}}_1={\mathbf{e}}_3$, ${\mathbf{e}}_4\circ {\mathbf{r}}_3={\mathbf{e}}_5$, ${\mathbf{e}}_5\circ {\mathbf{r}}_2={\mathbf{e}}_6$, but ${\mathbf{e}}_4\circ {\mathbf{r}}_1\ne {\mathbf{e}}_6$, which means the composition of relation ${\mathbf{r}}_2$ and ${\mathbf{r}}_3$ is noncommutative. Since ${\mathbf{r}}_2\circ {\mathbf{r}}_3={\mathbf{r}}_3\circ {\mathbf{r}}_2$ (the multiplication of complex numbers satisfies the commutative law), we can get ${\mathbf{e}}_4\circ {\mathbf{r}}_1={\mathbf{e}}_6$ in the same way as TransE, which contradicts the condition. So RotatE can not model noncommutative composition patterns. If we replace the condition ${\mathbf{e}}_4\circ {\mathbf{r}}_1\ne {\mathbf{e}}_6$ with ${\mathbf{e}}_4\circ {\mathbf{r}}_1={\mathbf{e}}_6$, then the composition of relation ${\mathbf{r}}_2$ and ${\mathbf{r}}_3$ becomes commutative composition. In this case the previous contradiction disappears, which means RotatE can model commutative composition patterns.

6. Experiments

In this section, we first evaluate QRSE with RotatE on a small knowledge graph made up of two families. This experiment will verify the superiority of QRSE in modeling noncommutative composition relation patterns. Then we evaluate QRSE and compare it with many baselines in two well-established and widely used real-world datasets.

6.1. Experiment on a KG about Two Families

There are 10 entities and 4 relations in the training KG. Each entity is a member of one family, and each relation is a type of kinship. Such as triple (Am1, son, Am2) means Am1 has a son called Am2. All of the triples in the training KG are shown in Figure 2, where each directed edge represents a triple, and its direction is from the head entity to the tail entity. Furthermore, the test set contains two triples: (Bm1, daughter_of_son, Bw3) and (Bm1, son_of_daughter, Bm3). We let models predict the head or tail entity for each test triple, so there are 4 queries during the test process.

Since we need 2 and 4 real numbers to determine a complex number and a quaternion respectively, we take ${\mathbb{C}}^{10}$ (i.e., embedding dimension $k=10$) and ${\mathbb{H}}^5$ (i.e., $k=5$) as the entity embedding spaces for RotatE and QRSE. Thus in practice, we can express the entity embeddings of RotatE and QRSE as 20-D real vectors. Except for the embedding dimension k, we keep other hyperparameters the same for two models: batch size $b=10$, self-adversarial sampling temperature $\alpha =0$, fixed margin $\gamma =0$, learning rate $\eta =0.001$, negative sampling size $\left|\mathcal{N}\right|=2$, and the order of norm in score function $p=2$.

We use Hit@1 to measure the performance of models, which means the proportion of the correctly answered queries (i.e., the true answer’s score is ranked first) among all test queries. The test performances of RotatE and QRSE are shown in Figure 3. We can see that QRSE gets the best Hit@1 value 1.00 quickly, and after that, it keeps this Hit@1 value all the time during the training process. RotatE also gets the best Hit@1 value quickly; however, after that, it’s Hit@1 value is always fluctuating between 0.5 and 1.00 randomly. To explain this phenomenon, we inspected the detailed scores and embeddings at step 16,000, which is large enough to ensure the convergence of the two models.

The top 3 scores for all test queries are shown in

Table 3. The detailed test results of RotatE and QRSE at training step 16,000.

Model	Test Triple	Entity to Predict	Ranked Scores for Top 3 Candidate Entities
RotatE	(Bm1, son_of_daughter, Bm3)	Bm1	Bm1: −0.0224, Am2: −5.4026, Bm3: −5.6849
		Bm3	Bm3: −0.0224, Bw3: −0.0227, Bm1: −5.6840
	(Bm1, daughter_of_son, Bw3)	Bm1	Bm1: −0.0178, Am2: −5.3975, Bm3: −5.6668
		Bw3	Bw3: −0.0178, Bm3: −0.0179, Bm1: −5.6654
QRSE	(Bm1, son_of_daughter, Bm3)	Bm1	Bm1: −0.00087, Aw2: −4.8535, Aw3: −4.9947
		Bm3	Bm3: −0.00087, Aw3: −5.2716, Bw3: −5.3903
	(Bm1, daughter_of_son, Bw3)	Bm1	Bm1: −0.00094, Aw2: −4.8532, Aw3: −4.9948
		Bw3	Bw3: −0.00094, Bm3: −5.3905, Am2: −5.8095

. For the two queries to predict the head entity Bm1, the scores of Bm1 are much higher than the second candidate entities for both RotatE and QRSE. However for the two queries to predict the tail entities Bm3 and Bw3, only QRSE keeps the large gap between the first and the second score, whereas RotatE gives very close scores for the top 2 candidate entities on both of the two queries. This result reveals that, for RotatE, the score ranks for the top 2 candidates are unstable and easily affected by the random noise on the two queries to predict the tail. That is why the Hit@1 of RotatE fluctuates during training. Moreover, for RotatE, the top 2 candidate entities are Bm3 and Bw3 for both of the two tail queries. Thus we guess the embeddings of these two entities are also very close.

Figure 4 shows the embeddings of Bm3 and Bw3 in RotatE and QRSE. As we guessed, the two embeddings are very close in RotatE but different in QRSE. This result verified that RotatE is unable to model noncommutative composition patterns, but QRSE can. Let us use the bold type to indicate the embeddings as before. For RotatE, along with the training process, Bw3 will close to $\mathbf{Bm}\mathbf{2}\circ \mathbf{daughter}$, and Bm2 will close to $\mathbf{Bm}\mathbf{1}\circ \mathbf{son}$. Hence Bw3 will close to $\mathbf{Bm}\mathbf{1}\circ \mathbf{son}\circ \mathbf{daughter}$. Similarly, Bm3 will close to $\mathbf{Bm}\mathbf{1}\circ \mathbf{daughter}\circ \mathbf{son}$. Because $\mathbf{daughter}\circ \mathbf{son}=\mathbf{son}\circ \mathbf{daughter}$, Bw3 will close to Bm3. For QRSE, $\mathbf{daughter}\otimes \mathbf{son}\ne \mathbf{son}\otimes \mathbf{daughter}$ in general, so Bw3 will not close to Bm3.

We can also show this fact by directly inspecting the relation embeddings of the two models in Figure 5. Note that $\mathtt{daughter}\oplus \mathtt{son}$ and $\mathtt{son}\oplus \mathtt{daughter}$ are not the relations in KG but the combinations made up of the relations in KG. Their “embeddings” are calculated from the embeddings of some relations (e.g., the “embedding” of $\mathtt{daughter}\oplus \mathtt{son}$ is $\mathbf{daughter}\otimes \mathbf{son}$ in QRSE). Obviously, the embeddings of son_of_daughter and daughter_of_son are almost the same in RotatE, since they are both approaching $\mathbf{daughter}\circ \mathbf{son}$ during training. However, they are different in QRSE since the embeddings of son_of_daughter is approaching $\mathbf{daughter}\otimes \mathbf{son}$ while the other is approaching $\mathbf{son}\otimes \mathbf{daughter}$ during training.

6.2. Experiment on Real-World Datasets

6.2.1. Experimental Setting

We still evaluated our method on two well-established and widely used real-world knowledge graphs, FB15k-237 [16] and WN18RR [17], with several strong baselines.

FB15k-237 is selected from FB15k [10], which is a subset of Freebase and mainly records the facts about movies, actors, and sports. Because FB15k suffers from test leakage through inverse relations: there are too many inversion patterns in KG, which are too easy to model, and even a simple rule-based model can perform well [17]. To make the results more reliable, FB15k-237 removed these inverse patterns. The statistics of FB15k-237 are 14,541 entities, 237 relations, 272,115 training triples, 17,535 validation triples, and 20,466 test triples.

WN18RR is selected from WN18 [10], which is a subset of WordNet and records lexical relations between words. WN18 also suffers from test leakage through inverse relations, so WN18RR removed its inverse patterns too. The statistics of WN18RR are 40,943 entities, 11 relations, 86,835 training triples, 3034 validation triples, and 3134 test triples.

The ranges of the hyperparameters for the grid search are following RotatE as embedding dimension $k\in \{125,250,500,1000\}$, batch size $b\in \{512,1024,2048\}$, and fixed margin $\gamma \in \{3,6,9,12,18,24,30\}$. Moreover, we searched self-adversarial sampling temperature $\alpha$ in $\{0.5,1.0,1.5\}$, learning rate $\eta$ in $\{0.00005,0.0001,0.0002\}$, negative sampling size $\left|\mathcal{N}\right|$ in $\{16,32,64,128\}$, and the order p of norm in score function in $\{2,3,4,5,6,7\}$. The embeddings are also uniformly initialized.

From each test triple (h, r, t), we generate two queries: (?, r, t) and (h, r, ?). Given each query, we can make a candidate triple by placing a candidate entity on the place of the entity to predict. The score of each candidate entity is just the score of its corresponding candidate triple. While ranking all the scores, we omit the scores of those candidate triples that already exist in training, validation, and test set, except the true answer for the query. This process is called “filtered” in some literature and is widely adopted in existing methods to avoid possibly flawed evaluation.

6.2.2. Results

We adopt these standard evaluation measures for both of the datasets: the mean reciprocal rank of the true answers (MRR), the proportion of queries whose true answers are ranked in the top k (Hit@k).

The link prediction results on real-world datasets are shown in

Table 4. Link prediction results on the FB15k-237 and WN18RR datasets. Numbers in boldface are the best, and underlined numbers are the second best.

Model	FB15k-237				WN18RR
	MRR	Hit@1	Hit@3	Hit@10	MRR	Hit@1	Hit@3	Hit@10
TransE	0.294	-	-	0.465	0.226	-	-	0.501
DistMult	0.241	0.155	0.263	0.419	0.43	0.39	0.44	0.49
ComplEx	0.247	0.158	0.275	0.428	0.44	0.41	0.46	0.51
ConvE	0.325	0.237	0.356	0.501	0.43	0.40	0.44	0.52
RotatE	0.338	0.241	0.375	0.533	0.476	0.428	0.492	0.571
DualE	0.330	0.237	0.363	0.518	0.482	0.440	0.500	0.561
DihEdral(STE)	0.320	0.230	0.353	0.502	0.480	0.452	0.491	0.536
DihEdral(Gumbel)	0.300	0.204	0.332	0.496	0.486	0.442	0.505	0.557
QuatE	0.311	0.221	0.342	0.495	0.481	0.436	0.500	0.564
QuatE(TC)	0.348	0.248	0.382	0.550	0.488	0.438	0.508	0.582
QRSE	0.350	0.252	0.390	0.548	0.491	0.443	0.508	0.581

. The result of TransE is taken from [29]. The results of DistMult, ComplEx, and ConvE are taken from [17]. The results of RotatE and DualE are taken from [13,27], respectively. The results of DihEdral(STE) and DihEdral(Gumbel) are taken from [28], where STE and Gumbel are two special treatments of the discrete relation embeddings. The results of QuatE and QuatE(TC) are taken from [14], where TC indicates the corresponding model using type constraints [36]. From this table, we can see that QRSE outperforms RotatE largely on all datasets and evaluation measures. This result supports our analysis of the modeling ability of the composition patterns. Compared with DihEdral(STE) and DihEdral(Gumbel), we find QRSE outperforms both of them on the two real-world datasets, whereas DihEdral(STE) is better than DihEdral(Gumbel) on FB15k-237 and just the opposite on WN18RR. This means the performance of DihEdral is easily affected by special treatments, and DihEdral can not perform well on the two real-world datasets simultaneously. Compared with DualE and QuatE, we find QRSE outperforms both of them too. This means that among all methods using (dual) quaternions so far, QRSE has explored the greatest potential of the (dual) quaternion space in the implementation of knowledge graph embedding. Because type constraints [36] can integrate prior knowledge into various KGE models and can significantly improve their performance in link prediction tasks, QRSE and most baselines display the results without it for fairness except QuatE(TC). Surprisingly, we can even see that QRSE is superior to QuatE with type constraints overall slightly. The success on this unfair comparison further demonstrates the excellence of QRSE. Overall, our QRSE has reached the state-of-the-art in link prediction problem on real-world datasets.

7. Conclusions and Future Work

We proposed a novel knowledge graph embedding model QRSE based on quaternions. QRSE is a KGE model that can model the noncommutative composition patterns. Besides, it can also model many other relation patterns, such as symmetry/antisymmetry, inversion, and commutative composition patterns. We varified these properties by theoretical proofs and experiments. From the definition of the triple approximate equation of QRSE, we can easily see that QRSE is a generalization of RotatE. Conversely, in some special cases, QRSE will degenerate to RotatE. For example, the case when the coefficients of $\mathit{j}$ and $\mathit{k}$ are fixed as 0 for all quaternions in all embeddings, and the modulus of all quaternions in relation embeddings are fixed as 1. Before QRSE, QuatE has already generalized ComplEx through replacing the complex numbers with quaternions. However, QuatE only takes advantage of that quaternions are more expressive than complex numbers. While our method not only leverages the expression advantage but also exploits the noncommutative property of quaternion multiplication to model the noncommutative composition patterns. The results of experiments on real-world datasets show that QRSE reaches the state-of-the-art on the link prediction problem. For future work, our plan is to combine QRSE with deep models for natural language processing. With its help, we expect deep models to achieve higher accuracy on question answering tasks and make the model’s answers more interpretable.