Causal Reinforcement Learning for Knowledge Graph Reasoning

Abstract

Knowledge graph reasoning can deduce new facts and relationships, which is an important research direction of knowledge graphs. Most of the existing methods are based on end-to-end reasoning which cannot effectively use the knowledge graph, so consequently the performance of the method still needs to be improved. Therefore, we combine causal inference with reinforcement learning and propose a new framework for knowledge graph reasoning. By combining the counterfactual method in causal inference, our method can obtain more information as prior knowledge and integrate it into the control strategy in the reinforcement model. The proposed method mainly includes the steps of relationship importance identification, reinforcement learning framework design, policy network design, and the training and testing of the causal reinforcement learning model. Specifically, a prior knowledge table is first constructed to indicate which relationship is more important for the problem to be queried; secondly, designing state space, optimization, action space, state transition and reward, respectively, is undertaken; then, the standard value is set and compared with the weight value of each candidate edge, and an action strategy is selected according to the comparison result through prior knowledge or neural network; finally, the parameters of the reinforcement learning model are determined through training and testing. We used four datasets to compare our method to the baseline method and conducted ablation experiments. On dataset NELL-995 and FB15k-237, the experimental results show that the MAP scores of our method are 87.8 and 45.2, and the optimal performance is achieved.

1. Introduction

In recent years, knowledge graphs have been widely used in several fields. However, for various professional fields, the constructed knowledge graphs are usually sparse and the implicit relationship between entities has not been fully examined, resulting in a poor practical application effect. Therefore, it is necessary to complete the knowledge graphs; the typical method is knowledge graph reasoning.

Knowledge graph reasoning tasks include Knowledge Graph Completion (KGC), quality verification, path prediction, relationship reasoning, and conflict detection [1]. This paper mainly focuses on KGC and transforms the knowledge graph reasoning problem into a problem predicting the missing part of the triple. For example, as depicted in Figure 1, we obtain two paths based on the existing knowledge graph: “Christopher Nolan$\stackrel{\mathrm{Born} \mathrm{in}}{\to }$London” and “London$\stackrel{\mathrm{Located} \mathrm{in}}{\to }$Britain”. Our task is to complete the triple: (Christopher Nolan, nationality, ?). Since we do not have the path “Christopher Nolan$\stackrel{\mathrm{Nationality}}{\to }$British”, we use the dotted line to represent the edge of the relationship. From knowledge graph reasoning, the above two paths are equivalent to “Christopher Nolan$\stackrel{\mathrm{Nationality}}{\to }$British”. So, we complete the triple: (Christopher Nolan, nationality, British).

In the above reasoning process, we start from the Nolan node. For the relationship of “nationality”, the association degree of the edges linked to the Nolan node differs, highlighting that “Born in” is more relevant to finding a relationship of nationality than “Direct” and “Partner”. This is because humans have prior knowledge that “place of birth” is more likely to determine “nationality” than “director” and “partner”.

Reinforcement learning is applied to knowledge graph reasoning tasks owing to its adaptability to complex environments [2]. Reinforcement learning regards the above reasoning process as the following problem: finding a path connecting the known head entity and tail entity, modeling this problem as a sequential decision problem, and solving it using the method based on policy gradient [3]. Since the existing reinforcement learning models do not conduct in-depth research on the knowledge graph, this paper employs causal inference to analyze the knowledge graph and obtain prior knowledge, thereby improving inference performance. For instance, in Figure 1, we want to complete the triple (person, language, ?), and there are two paths in the knowledge graph: “person$\stackrel{\mathrm{Star} \mathrm{in}}{\to }$film $\stackrel{\mathrm{Country}}{\to }$Britain” and “person$\stackrel{\mathrm{Star} \mathrm{in}}{\to }$film $\stackrel{\mathrm{Language}}{\to }$English”. We obtain prior knowledge through causal inference, i.e., the relationship “language” in the second path is more related to the relationship in the triple to be completed than the relationship “country” in the first path. So, we obtain the importance of the relationship in the path and thus guide the path selection.

The major contributions of this work are as follows:

• We propose a prior knowledge generation method based on causal inference. Specifically, it is implemented through the relationship importance identification module. The generated prior knowledge is mainly used to measure the contribution of the relationship in the triple to be completed.
• We introduce a new method combining causal inference and reinforcement learning that applies to knowledge graph reasoning. Specifically, the prior knowledge is integrated into the control strategy of the reinforcement model, allowing the agent to select the relation more accurately in each step. Our model also applies to large-scale knowledge graphs.
• The experiments demonstrate that our method’s performance outperforms the current baseline methods in most cases. Additionally, we conduct ablation experiments to highlight that our method is more effective than solely employing reinforcement learning.

The remainder of this paper is organized as follows. Section 2 briefly introduces the related work. Section 3 presents the methodological details, and Section 4 conducts experiments and analyzes the experimental results. Section 5 summarizes this work and provides future research directions.

2. Related Work

2.1. Knowledge Graph Reasoning

The existing methods in the field of knowledge graph reasoning involve two categories: methods based on distributed representation and based on relational path [4]. Knowledge graph methods based on distributed representation predict the missing tail entities by learning the low-dimensional embedded representation of known entities and relationships in triples. The most common method is TransE, which was proposed by Borders et al. [5]. Since the TransE method can only deal with the one-to-one relationship problem, many subsequent methods evolved from it. For instance, Wang et al. [6] and Lin et al. [7] introduced hyperplanes and mapping matrices to deal with the many-to-one relationship problem and proposed the TransH method and TransR method, respectively. In addition, Ji et al. [8] introduced a dynamic matrix and proposed the TransD method to solve this problem. However, these improved models cannot model and simultaneously infer all kinds of relationships (symmetry and composition). Thus, Pasquale et al. [9] developed ConvE, a two-dimensional convolutional link prediction method. Furthermore, Trouillon et al. [10] proposed the ComplEx method, which introduced complex number space into knowledge representation. The model structure of ComplEx is relatively simple, since only Hermitian dot products are used.

The above methods cannot solve the problem of multi-step knowledge reasoning, and therefore, solutions based on the relational path have been developed. Such a traditional model is the PRA (Path Ranking Algorithm) [11], characterized by strong interpretability and automatic rules discovery. However, this model is ineffective in dealing with low-frequency relationships and low-connectivity graphs (sparse data). In addition, when the graph is large enough, path extraction is time-consuming. Thus, Xiong et al. [12] and Das et al. [13] introduced reinforcement learning into knowledge graph reasoning, considering the relational path selection problem as a Markov decision process with better reasoning effect and interpretability. On this basis, Wan et al. [14] proposed an improved multi-level reinforcement learning model, which encodes historical knowledge and action space, achieving appealing results. The attention mechanism introduced by Wang et al. [15] enables the reasoning process to be memorized, and the graph neural network can enhance the entity semantics. It should be noted that the models introduced above do not fully employ the knowledge graph. Therefore, the accuracy and efficiency of their methods can be further improved.

2.2. Casual Inference

Causal inference mainly refers to inferring the relationship between the cause-and-effect variables. Specifically, appropriate intervention on variable X will lead to a change in the distribution of variable Y, but intervention on Y will not lead to a change in X. In this case, X is the cause variable of Y, and Y is the effect variable of X. A causal inference model mainly solves two kinds of problems. One is the intervention problem, i.e., the variable is set to a certain value, and the causal relationship is analyzed according to the data distribution change. The other problem is the counterfactual problem, i.e., the probability that events A and B do not occur is inferred when events A and B occur [16,17,18,19,20]. Since the problem of relational path selection in knowledge graphs can be transformed into a counterfactual problem, this paper focuses on counterfactual problems [21].

Counterfactual reasoning generally comprises three steps: (1) Traceability: Updating the value of a noise term using evidence. This step accounts for a particular noise with evidence in mind; (2) Intervention: Modifying the original model, replacing the structural equation with counterfactual variables, and obtaining the modified model; (3) Prediction: The modified model and noise are used to re-estimate the outcome variables, and the counterfactual results are obtained [22]. Some studies investigate the counterfactual problem, with counterfactual reasoning improving the reinforcement learning algorithms’ sample efficiency and interpretability [23]. For instance, Madumal et al. [24] proposed a behavioral influence model based on a structural causal model, which used the causal model for counterfactual analysis to improve the interpretability of the model. Lu et al. [25] developed a counterfactual-based data enhancement algorithm, which used the Structural Causal Model (SCM) to model the environment dynamics and estimate the causal effect based on the commonalities and differences in multi-domain data. Buesing et al. [26] introduced a CounterFactually Guided Policy Search algorithm CF-GPS, which counterfactually evaluates any policy based on SCM to improve policy performance and eliminate the model prediction bias. However, there are few studies on the combination of counterfactual problems and knowledge graph, and how counterfactual problems will help knowledge graph research remains to be explored.

2.3. Reinforcement Learning

Reinforcement learning is an important method in behavioral decision-making and control [27]. Specifically, the agent takes action at any time according to the next step’s current status and reward [28]. The way an agent chooses an action is called a policy, and reinforcement learning maximizes the cumulative reward by constantly improving the policy.

Reinforcement learning algorithms can be divided into model-based and model-free methods depending on whether the agent uses the dynamic model of the environment in policy updating. In the model-free reinforcement-learning method, the agent directly interacts with the environment and optimizes the policy end-to-end, which is easier to implement and has better asymptotic performance. Classic algorithms include SARSA (State-Action-Reward-State-Action) [29], Q-learning [30], deep Q-networks [31,32], and their variants [33]. Many studies have recently applied reinforcement learning to knowledge graph reasoning [34,35]. Indeed, Lin et al. [36] extended MINERVA and proposed the MultiHopKG model to solve the above problems. Wang et al. [14] proposed AttnPath to quantify the difficulty of relation learning through MSR and MRR and redesigned the model’s control strategy. At the same time, to avoid the stagnation of agents on the same entity, AttnPath designs a new learning mechanism that enables the agent to move forward at every step. Shen et al. [15] solved the problem of reward sparsity using a Monte Carlo search to improve the efficiency of knowledge reasoning. However, these models are highly data-dependent and perform poorly in non-stationary and heterogeneous scenarios.

2.4. Causal Reinforcement Learning

Causal reinforcement learning combines causal inference theory and reinforcement learning, allowing agents to understand the causal relationship and make correct decisions. Additional assumptions or prior knowledge is often required to achieve the above goals. Specifically, causal inference empowered reinforcement learning using improving sample efficiency, enhancing the generalization ability of agents, and reducing the impact of correlation on agents [37].

Maximilian et al. [38] measured the causal effect of an action on an object through constructing a causal quantity, thereby guiding the agent to explore the environment more effectively. Jiaxian et al. [39] enhanced the model’s generalization ability through measuring the controlled direct effect and reducing the encoding of the state irrelevant to the action into the context information. Pim et al. [40] believed that more information is not necessarily better, revealing the false causality’s misleading effect on reinforcement learning control strategy. Zhihong et al. [41] investigated the problem of false correlation in offline reinforcement learning and avoided uncertainty by pessimistic criteria to neglect the influence of false correlation on the model control strategy.

3. Proposed Method

This section describes the proposed model. First, the KGC problem is defined in detail. Then, the general framework of the proposed method is described, as well as how to extract prior knowledge through the causal inference module. Finally, the reinforcement learning framework is introduced.

3.1. Problem Definition

Let H denote the set of head entities, T is the entity found in the triple, and R represents the relation in the triple. G represents a known knowledge graph, $G=\{(h_i,r_i,t_i)\}\subseteq \{H,R,T\}$, $h_i\in H,r_i\in R,t_i\in T,i\in (1,n)$, where $i$ represents the number of steps the agent moved on the knowledge graph with $h_q$ as the starting point and the target entity as the endpoint. Our mission is to find the missing tail entity $t_q$ given the triple $(h_q,r_q,?)$. The proposed method solves the KGC problem based on a relational path. Therefore, we must find a path to the target tail entity in the knowledge graph, i.e., our goal is to find such a path denoted $p=\{h_q,r_1,r_2,\cdots ,r_n,t_q\}$. This goal is equivalent to finding a path with semantics equal to $r_q$.

3.2. Overall Architecture

Figure 2 illustrates our method’s framework. First, according to the triple $(h_q,r_q,?)$ and the path extracted from the knowledge graph, the prior knowledge is obtained through the counterfactual theory in causal inference. The prior knowledge is represented using the relationship weight in each path relative to the relationship to be queried and obtained using the relationship importance identification module. The relationship importance identification module will be described in detail in the next section. After obtaining the prior knowledge, we combine it with the policy network to guide the agent to make the right choice. The policy network mainly uses a Long Short-Term Memory (LSTM) to represent semantic information, which will be described later. Then, the reasoning step and the tail entity to be queried will be obtained, and finally, the knowledge graph will be updated.

3.3. Relationship Importance Identification

Acquiring prior knowledge from the knowledge graph can assist in efficiently selecting the path closest to the real answer from the relational paths and better completing the missing tail entity. We introduced that prior knowledge is represented by the weight of the relationship in each path relative to the relationship to be queried. In this way, we quantify the importance of each relational path in the knowledge graph. This weight is implemented through a counterfactual question in causal inference. Specifically, for triple $(h_q,r_q,t_q)$ with multiple paths from $h_q$ to $t_q$, we obtain the following conclusion through the counterfactual definition: when the relation $r_q$ changes, the tail entity will also change.

For the example in Figure 3, for the relationship $R_0$, if we change $r_2$ to $r_4$ in the path, and the rest remains unchanged, the relationship $R_0$ changes to $R_6$. Thus, we can obtain prior knowledge, as $r_2$ is more important for the relationship $R_0$. Then, we record this prior knowledge and the corresponding relationship individually.

Next, we construct a table of prior knowledge to represent the importance of the relationship. We denote this table by f, where r_i represents the relationship between the currently pointed entity and the next-hop entity after the agent moved $i$ steps and the number of times we obtain the prior knowledge that “r_i is important for $r_q$” by $f[r_q][r_i]$. Thus, the weight of r_i to $r_q$ is:

$${\omega }_R[r_q][r_i]=\frac{f[r_q][r_i]}{{\displaystyle {\sum }_{r_i\in R}f[r_q][r_i]}}$$

(1)

where ${\omega }_R$ denotes the set of weights and the set of extracted prior knowledge. A larger weight of r_i indicates that r_i should be selected first when selecting a path.

Similarly, we use ${\omega }_E$ to represent the weight of the entity:

$${\omega }_E[r_q][e_i]=\left\{{\omega }_R[r_q][r_j]|(e_i,r_j,e_k)\in G,e_k\in T\right\}$$

(2)

where ${\omega }_E[r_q][e_i]$ is the importance of the edge from entity $e_i$ concerning relation $r_q$.

Figure 3 depicts the structure of the relationship importance identification module based on causal inference. Since we have obtained the knowledge graph, it is easier to obtain prior knowledge through counterfactual methods in causal inference. In contrast to graphical models that require negative sampling, the proposed model only requires the correct triples, thus avoiding the noise introduced by false negative samples due to negative sampling.

3.4. Reinforcement Learning Framework

This section describes how to design a reinforcement learning model with the relationship importance identification module. Compared with the traditional reinforcement learning framework, we introduce prior knowledge to enhance the performance of our method. Specifically, we transform the path selection problem into a partially observable Markov decision process. Suppose the following triple $(h_q,r_q,?)$ is looked up in the knowledge graph. The agent starts from $h_q$ and chooses a relation according to the inference problem, thus jumping to the new entity. Based on the new entity jumped to, we continue to select the relationship and iterate several steps until we find the ultimate goal. In this process, the agent prefers the most promising relationship for state transition according to the optimization process and updates the policy network through the reward function to find the same correct path as the $r_q$ semantics.

3.4.1. Environment

Next, we represent the environment using five elements: (S, O, A, P, R).

S (State). The agent moves until it finds the target entity, so the state space at time t can be expressed in the following terms. One is the head entity $h_q$, which is the starting position of the agent. The second is the relation $r_q$, which is known in triples, and it is necessary to find a path in the knowledge map that is semantically equal to the relation $r_q$. The tail entity $t_q$ is the target to be queried, which can test the accuracy of the inference algorithm. The third is entity $e_t$, representing the entity the agent is pointing to. According to the description of the above three parts, we can express the state space as: $s_t=(h_q,r_q,t_q,e_t)$.

O (Optimization). The optimization aims to quantify the prior knowledge obtained using the relationship importance identification module, i.e., to express the importance of the relation as a weight value. Taking entity $e_t$ as the starting point, the prior knowledge is expressed as ${\omega }_E[r_q]$, and the optimization function is defined as $o_t=O(s_t=(h_q,r_q,t_q,e_t))=(h_q,r_q,e_t,{\omega }_E[r_q])$.

A (Actions). The action space is the set of all edges from the current entity to the next entity. Therefore, the action space can be expressed as $A_t=\left\{(r_{t+1},e_{t+1}|(e_t,r_{t+1},e_{t+1})\in G)\right\}$. Based on this, we represent the case where the inference terminates, which can be represented using an edge pointing to the entity.

P (Transition). The state transition space refers to the transition of the relationship path between the agent and the head entity and the transition of the current entity after the agent moves from one entity to another. The target entity to be searched remains unchanged during the agent’s movement. Thus, the state transition space can be expressed as: $P(S_t,A_t)=P((h_q,r_q,t_q,e_t),(r_{t+1},e_{t+1}))=(h_q,r_q,t_q,e_{t+1})$.

R (Rewards). Agents must constantly select actions from the action space, move forward in the knowledge map, and finally find the target entity. Therefore, we define the reward in the following form:

$$R=\left\{\begin{array}{cc}1& \hfill \mathrm{if} e_T=t_q\\ 0& \hfill \mathrm{otherwise}\end{array}\right.$$

(3)

where T denotes the last time the agent moves on the knowledge graph.

3.4.2. Policy Network

We use the policy network to guide the agent to make the right choice in the action space. The policy network is constructed using the prior knowledge extracted using causal inference and the neural network. Specifically, we first set a standard value $\phi$, while the current time is t, and the entity pointed by the agent is $e_t$. The agent obtains the weight value of each edge from the optimization space and compares it with $\phi$. If $\max ({\omega }_E[r_q][e_t])$ is larger than $\phi$, the edge with the largest value is selected from the optimization space. Otherwise, the corresponding edge is selected according to the reinforcement learning model.

When the neural network selects an action, a path memory component (LSTM) is considered to encode the path information so that the history exploration is memorized [42]. History embedding $h_t$ represents encoding the path that the agent moves on the knowledge graph before time t. The path is the composition of a series of entities and relationships, which can be represented as $\left\{h_q,r_1,e_1,r_2,e_2,...,r_t,e_t\right\}$. Then, $h_q$, $r_q$, and $e_t$ are concatenated and input into the nonlinear Multi-Layer Perceptron (MLP) [21]. The policy network $\pi$ is defined as follows:

$$\begin{array}{l}\pi (r_t|r_q,e_t,h_t,h_q)\\ =\left\{\begin{array}{c}{\omega }_E[r_q][e_t] \mathrm{if} \max ({\omega }_E[r_q][e_t])\ge \phi \hfill \\ soft\max (A_t(W_{2}RELU(W_1[h_t;h_q;r_q;e_t;a_t]))) \mathrm{otherwise}\end{array}\right.\end{array}$$

(4)

where $W_1$ and $W_2$ are weighting matrices.

3.4.3. Training

During training, according to the above strategy network, we update the parameter $\phi$ to maximize the expected cumulative reward $J$:

$$J(\phi )=E_{(e_1,r,e_2)\sim D}{E}_{a_1,a_2,...,a_{T-1}\sim \pi }[R(s_T|h_q,r_q)]$$

(5)

where $D$ represents all real and observed triples, the first E is the empirical average over the training dataset, and $s_T$ is the entity to which the agent is ultimately directed. The gradient update formula of the policy network can be expressed as follows:

$$\frac{\partial J(\phi )}{\partial \phi }=\frac{\partial }{\partial \phi }\log \pi (r_t|r_q,e_t,h_t,h_q)$$

(6)

The specific training steps are described in Algorithm 1. First, the parameter $\theta$ (Line 01) is initialized. Next, for each episode (Line 02), a path p (Line 03) is created, and the history embedding $h_0$ is initialized to 0 (Line 04). Next, for the historical path of the current node (Line 05), the agent calculates the history embedding, trains the policy network, and obtains the next action vector (Line 06 and Line 07). Next, the environment will transition to the next state and reward the agent for its choice (Line 08). By continuously using the formula to update $\phi$ (Line 09), the relationship path is determined to obtain the location of the target entity.

Algorithm 1. Training Procedure
01	Initialize $\phi$
02	for episode$\leftarrow$1 to N do
03	Initialize path p
04	Initialize entity representation $h_0$
05	for  $j\leftarrow 1 \mathrm{to} \left|p\right|$ do
06	Calculate history embedding $h_t$$\leftarrow$$LSTM(h_{t-1},[a_{t-1};r_q;h_q;e_t])$
07	Obtain reward R(p)
08	$\nabla J\leftarrow \frac{\partial }{\partial \phi }\log \pi (r_t|r_q,e_t,h_t,h_q)$
09	Update $\phi$
10	end for
11	end for

4. Experiments

4.1. Datasets

This work employs several classes of data sets that are widely used in knowledge graph reasoning, including (1) FB15K-237 [43], (2) WN18RR [44], (3) NEL-995 [45], (4) YAGO3-10 [46].

Table 1. Detailed of datasets FB15k-237, WN18RR, YAGO3-10 and NELL-995.

Dataset	E	R	Training Triples	Test Triples
FB15K-237	14,541	237	272,115	20,466
WN18RR	40,943	11	86,835	3134
NELL-995	75,492	200	154,213	3992
YAGO3-10	123,182	37	1,079,040	5000

reports the details of the four datasets.

FB15K-237 is a subset of FB15K, while FB15K is derived from the Freebase dataset. Freebase is a huge, multi-domain dataset created by Google, which collects a large amount of entity, attribute, and relationship information. The WN18RR dataset is derived from the WN18 dataset, and the WN18 dataset is derived from the WordNet dataset. Compared with the WordNet dataset, the WN18RR dataset removes the inversion relationship, retains other relationships, and has fewer relationship types. The NELL-995 dataset is an open-source machine learning dataset developed by OpenAI, which is constructed by unsupervised learning and automatically extracts entities and relationships through semantic analysis and pattern recognition of massive web content. YAGO3-10 is a subset of the YAGO dataset, with data from open-source websites or datasets such as Wikipedia, WordNet, and GeoNames.

4.2. Baseline Methods

We use typical representatives of methods based on distributed representation and based on a relational path as the baseline. The former methods comprise TransE, ComplEx, and ConvE, and the latter comprises PRA, DeepPath, and MINERVA.

4.3. Evaluation Protocol

We mainly evaluate the performance of each method through the following two tasks:

Relation Link Prediction. This is one of the common tasks to evaluate the reasoning performance of knowledge graphs. Specifically, first, we construct incomplete triples as negative samples, and second, we construct a test dataset containing both positive and negative samples. Then, we employ the Mean Average Precision (MAP) score for the specific evaluation index, which considers both the prediction accuracy and the relative position difference in the sequence.

Entity Link Prediction. This is also one of the common tasks used to evaluate the performance of knowledge graph reasoning. Specifically, we used Hit@N and Mean Reciprocal Rank (MRR). For the value of N in Hit@N, we take N as 1, 3, and 10 to obtain a more comprehensive evaluation result. The specific indicator should be selected in combination with the baseline method.

4.4. Implementation Details

In the experiment, we also need to determine the path length, the number of hidden layers of the neural network, the learning rate, the standard value, and other hyperparameters. We finally choose a path length from 2 to 6, the number of hidden layers from 50 or 100, and the learning rate from 0.01 to 0.05. The standard values are determined based on prior knowledge of each dataset. In addition, the training epochs must also be determined according to different data sets, ranging from 300 to 3000.

4.5. Results

This paper conducts relational link prediction tasks on the NELL-995 and FB15K-237 datasets.

Table 2. The MAP scores on NELL995 and FB15K-237.

Dataset	Tasks	TransE	PRA	DeepPath	MINERVA	Our Model
NELL-995	AthletePlaysInLeague	77.3	84.1	92.7	95.2	95.4
	AthletePlaysForTeam	62.7	54.7	75.0	82.4	86.9
	AthleteHomeStadium	71.8	85.9	89.0	89.5	90.7
	TeamPlaySports	76.1	79.1	73.8	84.6	85.5
	OrgHeadquaterCity	62.0	81.1	79.0	94.5	93.6
	BornLocation	71.2	66.8	75.7	79.3	82.2
	OrgHiredPerson	71.9	59.9	74.2	85.1	86.4
	PersonLeadsOrg	75.1	68.1	75.5	83.0	81.4
	…
	Overall	71.0	72.5	79.4	86.7	87.8
FB15k-237	adjoins	68.4	41.8	69.1	71.8	60.1
	contains	56.7	32.5	39.8	41.5	40.8
	personNationality	44.2	42.1	52.8	62.1	42.9
	filmDirector	41.5	32.8	45.6	38.9	46.3
	filmLanguag	61.5	45.1	52.5	58.9	46.1
	filmWritten	56.1	32.1	36.5	59.1	36.2
	capitalOf	42.5	25.8	43.8	48.9	50.6
	musicianOrigin	38.2	18.5	23.7	23.8	38.7
	…
	Overall	45.3	31.5	39.8	42.3	45.2

Bold indicates the highest score.

reports the MAP scores of TransE, PRA, DeepPath, MINERVA, and our model, highlighting that although our model does not perform optimally on every sub-task, it outperforms the baseline model on most tasks, especially on the NELL-995 dataset. Indeed, most individual tasks and the final mean scores were better than the baseline model. Although the highest scores were achieved on only three sub-tasks, the average scores of our model were also very close to the highest average scores on the FB15k-237 dataset.

The entity link prediction task was conducted on the NELL-995, WIN18RR, FB15K-237, and YAGO3-10 datasets.

Table 3. Link prediction results (Hits@N and MRR) on datasets WN18RR, FB15k-237, NELL-995 and YAGO3-10.

Dataset	Metric	TransE	ComplEX	ConvE	MINERVA	Our Model
NELL-995	Hit@1	0.514	0.612	0.672	0.663	0.684
	Hit@3	0.678	0.761	0.808	0.773	0.797
	Hit@10	0.751	0.827	0.864	0.831	0.871
	MRR	0.456	0.694	0.747	0.725	0.740
FB15K-237	Hit@1	0.248	0.303	0.313	0.217	0.228
	Hit@3	0.401	0.434	0.457	0.329	0.464
	Hit@10	0.450	0.572	0.600	0.456	0.491
	MRR	0.361	0.394	0.410	0.293	0.422
WN18RR	Hit@1	0.289	0.382	0.403	0.413	0.440
	Hit@3	0.475	0.433	0.452	0.456	0.485
	Hit@10	0.560	0.480	0.519	0.513	0.542
	MRR	0.359	0.415	0.438	0.448	0.474
YAGO3-10	Hit@1	0.248	0.260	0.350	0.355	0.380
	Hit@3	0.412	0.405	0.490	0.498	0.510
	Hit@10	0.315	0.550	0.620	0.650	0.678
	MRR	0.308	0.360	0.440	0.470	0.492

Bold indicates best performance.

reports the corresponding results, which reveals that our model performs better on all four datasets compared to the baseline models. Specifically, our model performs best in Hit@1 and Hit@10 and is second only to the ConvE model in Hit@3 and MRR on the NELL-995 dataset. This is because the ConvE model increases the embedding dimension by convolution, making it better at capturing complex information. Our model generally performs well on the FB15k-237 dataset because the FB15k-237 dataset has the most relationship types and fewer relationship patterns, and the model based on the relational path has an insufficient generalization ability. On the WN18RR and YAGO3-10 datasets, the relational path-based model outperforms the distributed representation-based model, and our model performs best among them. This is because prior knowledge contains more effective information.

4.6. Analysis

4.6.1. Setting of Standard Values and Ablation Study

We take the experimental results on the NELL-995 dataset as an example to study the influence of different standard values and further measure the influence of prior knowledge introduced by causal inference. Since the maximum weight of most relationships on the NELL-995 dataset is 0.1, we chose three standard values of 0.01, 0.05, and 0.1 for the following experiment. Furthermore, to ensure the consistency of the experimental results, we still use the Hits@N and MRR indicators to measure the performance differences in this ablation study, where N is still 1, 3, and 10. The experimental results are illustrated in Figure 4.

As depicted in Figure 4, after introducing the prior knowledge, the performance of the reinforcement-learning method is significantly improved, demonstrating our method’s effectiveness. It should be noted that the method’s performance is related to a standard value, as within a certain range, the larger the standard value, the better the method’s performance, preserving the standard value between 0.01 and 0.05. Beyond a certain range (for example, the standard value is between 0.05 and 0.10), the method’s performance decreases with the standard value’s increase. This is because setting the standard value too high may introduce noise. Therefore, it is necessary to set the standard value reasonably, according to different data sets, so that the model obtains the optimal performance.

4.6.2. Impact Analysis of the Number and Length of Paths

In the reasoning process, the agent will face the choice of multiple paths at each step, and the total number of these paths will impact the method’s performance. In addition, the length of the relationship path through which the agent reaches the target entity will also affect the method’s performance.

Therefore, we first analyze the impact of the total number of optional paths of agents on the method’s performance, taking the experimental results on dataset NELL-995 as an example. We first set the length of the agent’s relationship path to 4, then set the standard value to 0.1, and finally set the total number of agents’ optional paths to 3, 5, 10, and 50. In order to keep the consistency of the experimental results, we still use Hits@N and MRR to evaluate the method’s performance, where N is 1, 3, and 10. Figure 5 illustrates the corresponding results, which reveal that the method’s performance increases as the number of paths increases within a certain range (the number of paths is less than 5). Beyond a certain range (for example, the number of paths is greater than 10), the number of paths increases, and the method’s performance decreases. This may be due to errors in the knowledge graph or inaccurate prior knowledge obtained using the relationship importance identification module. Hence, the more prior knowledge is introduced, the greater the deviation is.

Next, we analyze the impact of path length on the method’s performance and conduct experiments on the dataset NELL-995. We fixed the number of paths to 5, set the standard value to 0.1, and set the path length to 2 to 6. The results in Figure 6 infer that within a certain range (path length less than or equal to 4), the method’s performance increases with the path length increase. The path length increases beyond a certain range (greater than or equal to 5), but the method’s performance is stable. This may be because the increase in path length brings more prior knowledge, which enhances the performance of the method, while the increase in path length also introduces more errors in the knowledge graph itself, which reduces the performance of the method, thus achieving a steady state.

5. Conclusions

This paper proposes a new method of knowledge graph reasoning based on causal reinforcement learning, as existing knowledge graph reasoning methods do not effectively use prior knowledge. Specifically, we use the counterfactual method in causal inference to obtain the weight of the relationship path, obtain the prior knowledge, and introduce it into the reinforcement learning reasoning model. The agent starts from the head entity and then selects the edge through the neural network when all the edge weights are small. Otherwise, it selects the edge according to the prior knowledge. Experiments on four datasets demonstrate the effectiveness of the prior knowledge. Follow-up research will focus on how to make the model automatically learn the standard value and how to effectively combine prior knowledge with semantic information to further improve the performance of reasoning.