A Keyword Detection and Context Filtering Method for Document Level Relation Extraction

Abstract

Relation extraction (RE) is the core link of downstream tasks, such as information retrieval, question answering systems, and knowledge graphs. Most of the current mainstream RE technologies focus on the sentence-level corpus, which has great limitations in practical applications. Moreover, the previously proposed models based on graph neural networks or transformers try to obtain context features from the global text, ignoring the importance of local features. In practice, the relation between entity pairs can usually be inferred just through a few keywords. This paper proposes a keyword detection and context filtering method based on the Self-Attention mechanism for document-level RE. In addition, a Self-Attention Memory (SAM) module in ConvLSTM is introduced to process the document context and capture keyword features. By searching for word embeddings with high cross-attention of entity pairs, we update and record critical local features to enhance the performance of the final classification model. The experimental results on three benchmark datasets (DocRED, CDR, and GBA) show that our model achieves advanced performance within open and specialized domain relationship extraction tasks, with up to 0.87% F1 value improvement compared to the state-of-the-art methods. We have also designed experiments to demonstrate that our model can achieve superior results by its stronger contextual filtering capability compared to other methods.

1. Introduction

RE is a crucial subtask in the field of information extraction. It can be divided into sentence-level extraction and document-level extraction according to the length of the input text. Previous work [1,2] mainly focused on predicting the relation between entities in the same sentence and could not handle cross-sentence extraction. The authors of [3] found that as many as 40.7% of the relational facts in the long corpus can only be extracted by combining the semantics of multiple sentences based on the statistics of the Wikipedia document corpus. Document-level corpus usually has a larger input length and a more complex structure, which poses great challenges to the information extraction capabilities of existing sentence-level models. Recently, many studies [3,4] have extended sentence-level RE to the document-level.

Many studies have adopted graph neural networks and used structure-dependent, heuristic, or structured attention mechanisms for relational reasoning [5,6,7,8,9]. The document graph they constructed can connect entities at a longer distance, which makes up for the shortcomings of the RNN-based encoder. With the introduction of the transformer model [10], studies have found that this model structure can implicitly capture long-distance dependencies [11,12]. Therefore, many studies have abandoned the graph neural network and used the pre-trained language model BERT [13] for relational reasoning and achieved good results.

However, previous research focuses on integrating global contextual information to enhance entity representation while ignoring the impact of genuinely critical local information on relation classification. As shown in Figure 1, we take the corpus mentioned in [14] as an example. When classifying the relation between “John Stanistreet” and “Bendigo” in the text, we mark their cross-attention in the pre-training model on the corresponding words. The correct classification relation of this entity pair should be place of birth and place of death. However, by observing the visualization results, we found that the pre-training model focuses on some entity nouns and punctuation marks that are unrelated to the classification [11]. The “born in” and “died in”, which are really helpful for the relation classification, were given little attention by the model. In addition, processing too much global context feature information may dilute the feature information provided by keywords, which will interfere and confuse the final classification task.

Based on the above observations, this paper proposes a keyword feature capture method based on the Self-Attention mechanism and uses the SAM module for recording context features in ConvLSTM [15]. By searching for the keyword embeddings with high cross-attention of entities, the local feature information that is helpful for relation classification is recorded to enhance the performance of downstream classification tasks. The contributions of this article are as follows:

• We present a BERT-LSTM-based model in the field of document RE that creatively uses the attention of subject-object entities to match the word embeddings that determine their relation;
• An improved ConvLSTM model is introduced to make it suitable for document sequence extraction. We build the model’s attention relation from scratch and enable it to extract useful contextual keyword features;
• Compared with state-of-the-art models, our model has achieved advanced performance on the three benchmark datasets, proving the effectiveness of using keywords for RE.

The rest of the article is organized as follows: Section 2 will describe the previous related work and the foundation of our work. Section 3 will explain how we decompose the RE task and present the overall structure of our method and the processing flow of each part. In Section 4, we report the experimental performance of our model and give a detailed analysis of the experimental results to demonstrate the effectiveness of the method. Section 5 summarizes the work with the limitations and future directions.

2. Related Work

Graph-based models and transformer-based models are the most common approaches for document-level RE. Graph-based methods are now widely adopted in RE because of their intuitiveness and effectiveness in relational reasoning. To cope with the challenge brought by document RE, the authors of [6] proposed an edge-oriented document-level RE graph neural model (EoG). The authors of [16] proposed a graphically enhanced dual attention network (GEDA) for attentional supervision from additional evidence to construct complex interactions between sentences and potential relational instances. The authors of [17] proposed a Dual-tier Heterogeneous Graph (DHG) to model the structural information of documents and perform relational reasoning across sentences. The authors of [18] proposed an encoder-classifier reconstructor model (HeterGSAN) that reconstructs the paths and puts more attention on related entity pairs instead of negative samples. In general, graph-based methods are popular in current applications and can overcome the problems caused by long-distance dependencies. However, the type of nodes and edges used in the graph-based approach requires delicate manual design, which is not considered elegant, and it is a common problem of the graph-based method on non-graph data.

Many researchers have also proposed the transformer-based approach. The transformer’s multi-headed attention component can divide semantic features into multiple subspaces, allowing the model to learn richer contextual features from the text. The authors of [19] observed an unbalanced relation distribution on the DocRED dataset. They proposed a two-step RE mode, separating relation recognition and classification, which can avoid the influence of a large number of negative samples in the dataset. The authors of [20] proposed a hierarchical inference network reasoning network (HIN) for document-level reasoning. It uses a hierarchical reasoning method to aggregate the reasoning information at the entity level, sentence level, and document level. The authors of [21] proposed CorefBERT, which aims to capture co-referential information in the text. They incorporated entity-pair matching, mention-entity matching, and mention-mention matching into the pre-trained model to enhance the representation of the relational extraction model. The authors of [14] proposed a model based on entity-enhanced embedding and adaptive threshold loss function. They used the attention matrix output by the pre-training model BERT to calculate the cross-attention between entities and introduced a threshold class to calculate the loss of positive and negative samples, respectively, which allowed their model to achieve SOTA results on multiple datasets. We can summarize the predecessors’ work into two aspects:

• How to construct potentially useful information for determining the existence of relations from a large amount of textual data;
• How to overcome the uneven distribution of relations in the dataset and the negative impact of a large number of negative samples on the model.

In addition, there have been research attempts to extract relations based on keywords in the field of professional documents. The authors of [22] used a domain keyword collection mechanism to guide the model to focus on the semantic interaction between biological entities linked by keywords by manipulating the attention mask, which improved the F1 value by 5.6% on the Biocreative V Chemical Disease Relation (CDR) dataset. The authors of [23] used the BiLSTM neural network and attention mechanism to extract time and sentence-level important information. They selected each sentence’s critical feature layer under the pooling layer’s action, which has also achieved advanced results in the field of patent document RE. In general, they constructed keyword-attentive models and made them more sensitive to specific words in the specialized domains. However, this approach cannot be generalized to open domains because it is not possible to enumerate all keywords in open domains.

The authors of [24] used a SALSTM model that can mine and capture remote dependencies in the spatiotemporal sequence prediction task in the field of computer vision and achieved the most advanced results on multiple tasks such as precipitation prediction. Similar to precipitation prediction, the RE task is also a sequence feature extraction and prediction task. Inspired by this, we propose a SAM-LSTM model that can be used to process document sequences, integrates a self-attention memory module and trains a new attention layer from scratch. See the next section for specific methods.

3. Methodology

The structure and processing flow of the method proposed in this paper are shown in Figure 2. The word embedding output by the pre-training model is the original input of the SAM-LSTM model, and the Self-Attention mechanism lets the model learn to find the critical feature information. The weighted feature after attention screening is combined with entity mentions and input to the fully connected layer for the final classification.

3.1. Encoder

We model the document-level RE task: with a document d, ${\left\{e_i\right\}}_{i=n}^n$ is the set of all entities it contained. The aim is to find out the existing relationship from the pre-specified relationship set $R\cup \left\{NA\right\}$(NA stands for no relation) for every pair of entity pairs $\left(e_s,e_o\right)$ in the set. An entity $e_i$ can be mentioned multiple times in the document as ${\left\{m_k^i\right\}}_{k=1}^{e_i}$. If the entity pair $\left(e_s,e_o\right)$ contains relation, then this relation is expressed through one of their mentions. The RE model needs to classify the relation between all entities in document d. In order to further improve the task performance, we adopt the BERT pre-training model and integrate the following technologies to build the entire model. We can think of a document as a sequence of words $d={\left[x_t\right]}_{t=1}^l$. We add the “*” symbol before and after the entity mention to mark the position of the entity. Then, we enter the document into the pre-training model BERT to obtain the word embedding sequence H of the entire document:

$$\mathit{H}=\left[h_1,h_2,\dots ,h_l\right]=BERT\left(x_1,x_2,\cdots ,x_l\right)$$

(1)

For some documents whose length exceeds 512, we utilize dynamic windows to encode the entire document, and average the embedding of overlapping marks in different windows to get its final representation. Multiple mentions of an entity may produce different word embeddings, which is not conducive to further classification processing. As a result, logsumexp pooling [25] is used in this paper. The embedding of the entity is obtained using the following formula for multiple mentions.

$$h_{e_i}=log\sum _{k=1}^{n_{e_i}}exp\left(h_{m_k^i}\right)$$

(2)

This pooling operation is similar to max pooling, but it can better accumulate the mentioned information, and it also shows better performance than average pooling in the experiment.

3.2. Context Feature Extraction

Since the data volume remains large after one document word embedding, this paper performs contextual feature processing based on the ConvLSTM model [15]. This model replaces the input-to-state, state-to-state fully connected operations of ordinary LSTM with convolutional operations, which can establish temporal relations as LSTM and carve local spatial features as CNN. It also enhances feature extraction while reducing the amount of operations. As shown in Figure 3, $h_i$ is the vector in the word embedding sequence, ${\mathit{C}}_i$ is the state corresponding to cell at step i, ${\mathit{H}}_i$ is the current input feature, ${\mathit{M}}_i$ is the sequence information accumulated by cell at step i.

This model sequentially processes the word embedding sequence $h_i$, controls the cell state ${\mathit{C}}_i$ by the operations of forgetting gate, input gate and output gate. Meanwhile, the SAM module is used to calculate the attention values with entity embeddings $h_{e_s}$ and $h_{e_o}$ for the memory ${\mathit{M}}_{i-1}$ and hidden state ${\widehat{\mathit{H}}}_{i-1}$. Using this attention value to weight the current input features and the features stored in the memory module, we finally output the ${\widehat{\mathit{H}}}_i$ and update the ${\mathit{M}}_i$.

3.3. Self-Attention Memory

It has been shown previously that obtaining the attentional relations from the pre-trained model may not yield good results for downstream relational classification tasks. We train a new attentional layer from scratch compared to the previous work [14] which directly uses the multiheaded attention of BERT to process the full-text context. In this paper, a novel SAM module is proposed for capturing key features of full-text, while a memory unit $\mathit{M}$ is used to store contextual sequence information with a global sensory field.

The main idea of the module processing is to use entity pairs to match the word embeddings of interest in the input sequence. Following ConvLSTM processing, the attention matrix of the entity pairs and features is generated, and the parts with greater attention are weighted and updated to be stored in the memory unit. This ensures that the model is able to store the key features of the entity’s dependency on the relation even after traversing the entire sequence of contexts.

The detailed structure of the SAM module is shown in Figure 4, where ${\widehat{\mathit{H}}}_{i-1}$ is the current hidden feature being processed by ConvLSTM, and ${\mathit{M}}_{i-1}$ is the memory accumulated since the previous step. ${\mathit{H}}_e$ is the entity pair matrix, which is connected by the subject-object entity embedding $h_{e_s}$ and $h_{e_o}$. The module can be divided into three parts: self-attentive feature aggregation, memory update, and output.

3.3.1. Feature Aggregation

To match the contexts of interest for entity pairs, we first splice ${\widehat{\mathit{H}}}_{i-1}$ and ${\mathit{M}}_{i-1}$ into the aggregation matrix ${\mathit{C}}_{i-1}$, and then perform a self-attentive operation on the entity pair matrix ${\mathit{H}}_e$ with the aggregation matrix ${\mathit{C}}_{i-1}$. First, we calculate the value matrix ${\mathit{C}}_{i-1}$ corresponding to ${\mathit{V}}_c=W_v{\mathit{C}}_{i-1}\in {\mathbb{R}}^{C\times N}$ and the key matrix ${\mathit{K}}_c=W_k{\mathit{C}}_{i-1}\in {\mathbb{R}}^{C\times N}$. Then, we calculate the query matrix corresponding to ${\mathit{H}}_e{\mathit{Q}}_e=W_q{\mathit{H}}_e\in {\mathbb{R}}^{C\times 2}$, where $W_v$, $W_k$, $W_q$ are the weight parameters of the 1 × 1 convolution kernel. The product of the matrices is then used to compute the attention matrix $\mathit{A}$ of the entity pair for the aggregated feature:

$$\mathit{A}=softmax\left({\mathit{Q}}_e^T{\mathit{K}}_c\right)\in {\mathbb{R}}^{2\times N}$$

(3)

The $a_{i=0,j}$ in $\mathit{A}$ represents the subject’s attention to the $j^{th}$ feature vector in ${\mathit{C}}_{i-1}$, $a_{i=1,j}\in \mathit{A}$ represents the object’s attention to the $j^{th}$ feature vector in ${\mathit{C}}_{i-1}$. To find the feature vector that both the subject/object entities are interested in, we calculate the cross-attention by multiplying the attention of both, and the normalized cross-attention weights are multiplied with the value matrix ${\mathit{V}}_c$ to get the new aggregated feature ${\mathit{C}}_i$. Finally, ${\mathit{C}}_i$ is stitched together with the current moment feature as the input for the next step:

$$\begin{array}{c}{a}^{(s,o)}=a_{i=0,j}·a_{i=1,j}j\in \{1,2,\dots ,N\}\\ a^{(s,o)}=a^{(s,o)}/{\mathbf{1}}^T{a}^{(s,o)}\\ {\mathit{C}}_i={\mathit{V}}_c^T{a}^{(s,o)}\\ \mathit{Z}=W_z\left[{\mathit{C}}_i;{\widehat{\mathit{H}}}_{i-1}\right]\end{array}$$

(4)

3.3.2. Memory Update

In this paper, a gating mechanism similar to the GRU [26] (Gate Recurrent Unit) model is used to update the memory adaptively. The input gating $i^{\prime }$ and the fusion feature $g^{\prime }$ are generated by aggregating the features ${\mathit{C}}_i$ and the original input ${\widehat{\mathit{H}}}_{i-1}$. To reduce the parameters, it is straightforward to use $\left(1-i^{\prime }\right)$ to represent the forgetting gate, and the specific update process is as follows. To reduce the number of parameters, we directly use $\left(1-i^{\prime }\right)$ to represent the oblivion gate. The specific update process is as follows:

$$\begin{array}{c}\begin{array}{c}{i}^{\prime }=\sigma \left(W_{m;zi}\ast {\mathit{C}}_i+W_{m;hi}\ast {\widehat{\mathit{H}}}_{i-1}+b_{m;i}\right)\\ g^{\prime }=tanh\left(W_{m;zg}\ast {\mathit{C}}_i+W_{m;hg}\ast {\widehat{\mathit{H}}}_{i-1}+b_{m;g}\right)\\ {\mathit{M}}_t=\left(1-i^{\prime }\right) \circ {\mathit{M}}_{t-1}+i^{\prime } \circ g^{\prime }\end{array}\end{array}$$

(5)

To further reduce the computational effort, this paper uses depthwise separable convolution [27] instead of standard convolution operations. Compared with the original storage unit updated by ConvLSTM through convolutional operations, the memory module proposed in this paper is updated not only by convolutional operations but also by aggregating features ${\mathit{C}}_i$, which can capture the global dependencies of entities at all times. Therefore, we consider that the memory module can contain the international and contextual information filtered by the attention mechanism.

3.3.3. Output

The output of the module is the dot product of the output gate $o^{\prime }$ and the updated memory ${\mathit{M}}_i$. The procedure is as follows:

$$\begin{array}{c}{o}_t^{\prime }=\sigma \left({\mathit{W}}_{m;zo}\ast \mathit{Z}+W_{m;ho}\ast {\mathit{H}}_i+b_{m;o}\right)\\ {\mathit{H}}_i=o_t^{\prime } \circ {\mathit{M}}_t\end{array}$$

(6)

3.4. Classification Module

The entity pair embedding $e_s$, $e_o$ and the contextual key feature matrix ${\mathit{M}}^{(s,o)}$ can be obtained through the above steps for the word embedding sequence. These entities are mapped to hidden states by feedforward networks, and then, we use bilinear functions and sigmoid activation functions to calculate the probability of the relation r:

$$\begin{array}{cc}\hfill & \hfill z_s=tanh\left(W_s{e}_s+W_{m1}{\mathit{M}}^{(s,o)}\right)\hfill \\ \hfill & \hfill z_o=tanh\left(W_o{e}_s+W_{m2}{\mathit{M}}^{(s,o)}\right)\hfill \\ \hfill & \hfill \mathrm{P}\left(r\mid e_s,e_o\right)=\sigma \left(z_s{W}_r{z}_o+b_r\right)\hfill \end{array}$$

(7)

where $\left\{W_s,W_0,W_{m1},W_{m2}\right\}\in {\mathbb{R}}^{d\times d}$ are the training parameters of the model, $p\left(r\mid e_s,e_o\right)$ is the probability of the existence of various relations between entity pairs predicted by the model. Previous work [19] observed that the entity relation extraction dataset generally has an unbalanced relation distribution and most of the entity pairs are unrelated to each other. So, inspired by the idea that the target class score is greater than every non-target class in Circle Loss [28], we introduce a balanced loss function training model. Initially, a loss function in the logsumexp form can be formulated as follows:

$$log\left(1+\sum _{i\in {\Omega }_{pos,j},{\Omega }_{neg}}{e}^{s_i-s_j}\right)$$

(8)

We also note that the RE task is essentially a multi-label classification task. Therefore, a threshold is needed to determine which classes need to be output. Instead of introducing an additional threshold class as in the previous work [14], we directly use the score $s_{NA}$ of the unrelated class $NA$ as the threshold, and thus, the loss function can be further obtained as:

$$\begin{array}{cc}& log\left(1+\sum _{i\in {\Omega }_{pos,j}\in {\Omega }_{neg}}{e}^{s_i-s_j}+\sum _{i\in {\Omega }_{pos,j}\in {\Omega }_{neg}}{e}^{s_i-s_{NA}}+\sum _{i\in {\Omega }_{pos,j}\in {\Omega }_{neg}}{e}^{s_{NA}-s_j}\right)\hfill \\ \hfill =& log\left(e^{s_{NA}}+\sum _{i\in {\Omega }_{pos}}{e}^{s_i}\right)+log\left(e^{-s_{NA}}+\sum _{j\in {\Omega }_{neg}}{e}^{-s_j}\right)\hfill \end{array}$$

(9)

This loss expects all target class scores to be greater than each non-target class, and also expects all target class scores to be greater than the threshold $s_{NA}$ and all non-target class scores to be less than the threshold $s_{NA}$. In this way, the model can automatically learn and determine the labels to be output: output the relation classes greater than the threshold as having a relation, and otherwise output no relation label $NA$.

4. Experiment and Results

4.1. Dataset

We evaluated the proposed model on three popular document-level relationship extraction datasets (DocRED [4], CDR [29], and GDA [30]), all of which involve challenging relational inference on multiple entities across multiple sentences. DocRED and CDR provide human-validated data. They use crowdsourcing to manually annotate all possible relationships in the documents, which ensures the quality of the data and better guides the deep learning model. In order to ensure the validity and generality of our approach, we have compared performance primarily on the human-validated part of the DocRED dataset. The statistical information for each dataset is shown in

Table 1. Summary of DocRED, CDR and GDA datasets.

Dataset	Train	Dev	Test	Entities/Doc	Mentions/Doc	Mention/Sent	Relation
DocRED	3053	1000	1000	19.51	26.19	3.58	96
CDR	500	500	500	6.78	19.21	2.48	1
GDA	29,192	-	1000	4.80	18.53	2.28	1

.

DocRED is a large-scale dataset built with Wikipedia. It provides comprehensive manual annotation of entity mentions, entity types, relationship facts, and corresponding inference-supporting evidence. In addition, DocRED collects remotely supervised data, which uses a fine-tuned BERT model to identify entities and link them to Wikidata. 101,873 document instances are then scaled by obtaining relationship labels through remote monitoring.

CDR (Chemical-Disease Reactions dataset) is a biomedical dataset constructed with PubMed abstracts. It contains 1500 documents with human annotations, which are equally divided into a training set, a development set, and a test set. CDR is a binary classification task designed to identify induced relations from chemical entities to disease entities, and is of great importance for biomedical research.

GDA (Gene-Disease Associations dataset) is similar to the CDR and is also a binary relationship classification task for identifying interactions between gene and disease concepts but at a much larger scale. Constructed using the remotely supervised MEDLINE summarization technique, it contains 29,192 documents as training set and 1000 documents as a test set.

4.2. Experimental Settings

The experiments in this paper are based on Pytorch, using BERT-base and RoBERTa-large as Encoder on DocRED, and SciBERT-base as Encoder on CDR and GDA datasets. We use AdamW and set a warm-up learning rate of 6% to optimize the model. The detailed hyper-parameters settings are shown in

Table 2. Hyper-parameters Setting.

Method	DocRED		CDR	GDA
	BERT	RoBERTa	SciBERT	SciBERT
Batch size	4	4	4	16
Epoch	30	30	30	10
lr for encoder	5 × 10 ${}^{-5}$	5 × 10 ${}^{-5}$	2 × 10 ${}^{-5}$	2 × 10 ${}^{-5}$
lr for LSTM	1 × 10 ${}^{-4}$	1 × 10 ${}^{-4}$	1 × 10 ${}^{-4}$	1 × 10 ${}^{-4}$

. The model is trained on an NVIDIA P100 16GB GPU. We measured RE performance by calculating precision, recall and F-measurement scores on the test set. By evaluating all false positives, false negatives, correct positives and correct negatives, we used the standard formula to calculate the F1-score as:

$$\begin{array}{c}\mathrm{Precision}=\frac{\mathrm{correct}\mathrm{positive}\mathrm{predict}}{\mathrm{all}\mathrm{samples}\mathrm{predicted}\mathrm{as}\mathrm{positive}}\\ \mathrm{Recall}=\frac{\mathrm{correct}\mathrm{positive}\mathrm{predict}}{\mathrm{all}\mathrm{positive}\mathrm{samples}}\\ \mathrm{F}1=\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}\end{array}$$

(10)

4.3. Results on the DocRED Dataset

We compared our model with graph neural models, including GEDA [16], LSR [7], GLRE [9], GAIN [8], HeterGSAN [18]; and transformer-based models, including BERT [19], BERT-TS [19], HIN [20], Coref [21], ATLOP [14]. We run five experiments with different random seeds and record the mean and standard deviation on the development set. The experimental results are shown in

Table 3. Results (%) on the development and test set of DocRED.

Model	Year	Dev		Test
		Ign F1	F1	Ign F1	F1
GEDA-BERT${}_{base}$ [16]	2020	54.52	56.16	53.71	55.74
LSR-BERT${}_{base}$ [7]	2020	52.43	59.00	56.97	59.05
GLRE-BERT${}_{base}$ [9]	2020	-	-	55.40	57.40
GAIN-BERT${}_{base}$ [8]	2020	59.14	61.22	59.00	61.24
HeterGSAN-BERT${}_{base}$ [18]	2021	58.13	60.18	57.12	59.45
BERT${}_{base}$ [19]	2019	-	54.16	-	53.20
BERT-TS${}_{base}$ [19]	2019	-	54.42	-	53.92
HIN-BERT${}_{base}$ [20]	2020	54.29	56.31	53.70	55.60
CorefBERT${}_{base}$ [21]	2020	55.32	57.51	54.54	56.96
ATLOP-BERT${}_{base}$ [14]	2021	59.22	61.09	59.31	61.30
SAM-LSTM-BERT${}_{base}$		60.18 ± 0.14	61.96 ± 0.15	60.68	62.02
BERT${}_{large}$ [19]	2019	56.67	58.83	56.47	58.69
CorefBERT${}_{large}$ [19]	2020	56.82	59.01	56.40	58.83
RoBERTa${}_{large}$ [20]	2020	57.14	59.22	57.51	59.62
CorefRoBERTa${}_{large}$ [21]	2020	57.35	59.43	57.90	60.25
ATLOP-BERT${}_{large}$ [14]	2021	61.32	63.18	61.39	63.40
SAM-LSTM-RoBERTa${}_{large}$		62.13 ± 0.13	63.89 ± 0.16	62.25	64.03

, and the SAM-LSTM-BERTbase in this paper achieved the best-known results on the DocRED dataset. These models’ performance results are taken from their original articles.

4.4. Results on the Biomedical Datasets

In experiments on two biomedical datasets, we compared SAM-LSTM with many models, including EoG [6], LSR [7], DHG [17], GLRE [9], and ATLOP [14]. We applied the SciBERTbase [31] model pre-trained in the scientific publications corpus. The results are shown in

Table 4. Results (%) on the biomedical datasets CDR and GDA.

Model	Year	CDR	GDA
EoG [6]	2019	63.6	81.5
LSR [7]	2020	64.8	82.2
DHG [17]	2020	65.9	83.1
GLRE [9]	2020	68.5	-
SciBERT${}_{base}$ [31]	2019	65.1	82.5
Kw-BioBERT [22]	2021	76.4	-
ATLOP-SciBERT${}_{large}$ [14]	2021	69.4	83.9
SAM-LSTM-SciBERT${}_{large}$		73.5	84.7

, where SAM-LSTM improved the F1 scores of GDA 0.8%. We also note that [22] achieved better results than we did on the CDR dataset. We believe this is due to their use of a pre-trained biomedical language representation model and the injection of domain expertise in the RE process.

4.5. Ablation Study

A series of comparative experiments are conducted to validate the effectiveness of the components in this paper. This is shown in

Table 5. Ablation study of SAM-LSTM on Dev.

Model	Ign F1	F1
w/o Memory	59.45	60.89
w/o Self-Attention	58.88	60.48
w/o Balanced loss	59.32	61.22
w/o SAM-LSTM	57.60	59.52

.

w/o Memory means that the model only calculates self-attention with the current input each time and does not update memory; w/o Self-Attention means that the standard ConvLstm operation is used to replace the Self-Attention part of the model; w/o Balanced loss means that the loss function of the baseline is used. In addition, considering that the BERT model also comes with a multi-headed attention mechanism, we directly remove the LSTM layer and test the effect of the BERT+classifier.

The result reveals that the performance of the model decreases after removing any of the components. The Memory module and Self-Attention have the greatest impact, bringing a decrease of 1.07% and 1.48% after removal. This indicates the effectiveness of the SAM-LSTM network for extracting document-level entity relations. In addition, the results of LSTM removal demonstrate that directly training the attentional relations of BERT to weight the contexts cannot achieve the desired results. We suggest that this may be due to the large parameters of the pre-trained model, and the training corpus within the dataset cannot support the model to shift its attention to keywords through learning rapidly.

4.6. Case Study

In order to further investigate the effectiveness of this model compared to the ATLOP model, we divided the dataset into different groups in terms of corpus length and tested the ATLOP model and our model on them separately.

From the results in Figure 5, it can be seen that the SAM-LSTM model in this paper consistently outperforms the ATLOP model. It can also be seen that the performance of our model decreases more slowly when the text length rises. This indicates that the method in this paper can preserve the key feature information even in longer difficult cases through a powerful self-attentive mechanism, while being less affected by irrelevant information. We also note that the time consumption of our model does not increase significantly when faced with long difficult cases. This is mainly influenced by the length of the sequence due to the nature of the LSTM itself. However, the processing time of our method increased by a factor of 1.5 on average compared to ATLOP. We believe that this is mainly due to our strategy of successfully matching keywords for each pair of entities, which generates a relatively large amount of computation when there are more entities in the document.

In addition, to explore whether the models in this paper can actually capture key features, experiments are designed to compare the contextual feature extraction ability of the SAM-LSTM model with that of the ATLOP model. The analysis continues using the corpus mentioned in Section 1. While classifying the relation between the two entities “John Stanistreet” and “Bendigo” in the text, the contextual features used by the two models for classification are first obtained. Then, the cosine similarity is computed between the two and the word embeddings of the manually selected keywords “born” and “died”.

The experimental results in

Table 6. Keyword similarity comparison.

Keyword	ATLOP	SAM-LSTM
“born”	0.6360	0.6813
“died”	0.5669	0.6482

show that the similarity of our model is up to 8.13% higher than that of ATLOP. This supports why our model performs better: ATLOP uses the attention relation of the pre-trained model to weight the average word embedding of the full text, so that the key features are diluted by the large number of irrelevant vectors within the document, which deviates from the features originally needed for the downstream classification task. In contrast, the self-attentive memory unit used in this paper can capture and record these features while processing full-text information, which can achieve more favorable results for classification.

To further validate the performance of our model in difficult cases, we chose a text containing distant relational dependencies as an example. As shown in Figure 6, when our model classifies the relationship between “Tang Dynasty” and ’Chang’an’, it can be seen that the subject ’Tang Dynasty’ is separated from the object ’Chang’an’ by 13 sentences.

However, we can see from the similarity visualization that our model still accurately filters the keyword “capital” in 15 sentences and finally outputs features that are highly similar to it.

5. Conclusions

This paper proposes the SAM-LSTM model to reconstruct global entity dependencies to find the critical semantic features. We first illustrate that it is not appropriate to use the attention layer of the pre-trained model directly. Then, we construct the new keyword attention relation and extract contextual features via recording and updating memory. Finally, we use a multi-label classification loss function that can balance the categories. With the above techniques, we further improve the performance of the document-level RE task on a public dataset. This paper also designs experiments to demonstrate that the model in this paper can better capture local keyword information compared to the current state-of-the-art models. The approach of enhancing relation classification by capturing the keyword features of entity relation dependencies is effective. Since we have achieved advanced results on both open and specialized domains/datasets, we can say that the approach of this paper can be applied to the domains not exposed to the model. Without constructing keyword features for each specific domain, our model has strong applicability to the new domains. In the course of our experiments, we also found that processing the context for each entity pair is relatively time-consuming. We will explore a common keyword extraction method for documents in the future to speed up the computation of the model and allow all entity pairs to find the content of interest in a shared keyword pool.