Unlocking Everyday Wisdom: Enhancing Machine Comprehension with Script Knowledge Integration

Abstract

Harnessing commonsense knowledge poses a significant challenge for machine comprehension systems. This paper primarily focuses on incorporating a specific subset of commonsense knowledge, namely, script knowledge. Script knowledge is about sequences of actions that are typically performed by individuals in everyday life. Our experiments were centered around the MCScript dataset, which was the basis of the SemEval-2018 Task 11: Machine Comprehension using Commonsense Knowledge. As a baseline, we utilized our Three-Way Attentive Networks (TriANs) framework to model the interactions among passages, questions, and answers. Building upon the TriAN, we proposed to: (1) integrate a pre-trained language model to capture script knowledge; (2) introduce multi-layer attention to facilitate multi-hop reasoning; and (3) incorporate positional embeddings to enhance the model’s capacity for event-ordering reasoning. In this paper, we present our proposed methods and prove their efficacy in improving script knowledge integration and reasoning.

1. Introduction

In recent years, the rapid development of natural language processing (NLP) technologies has taken place. They have been widely applied into numerous everyday products such as search engines, smart assistants, mobile devices, and more. Very recently, large language models such as ChatGPT and Bard have ushered in a new era of NLP research and application, thereby showcasing their unprecedented performance results across many tasks and heir gaining extensive adoption by users worldwide.

Despite the significant achievements, the journey towards developing an NLP system with human-level cognitive abilities is far from over. One of the pressing challenges is the integration of commonsense knowledge into machine comprehension systems—an area which has seen growing attention [1]. Commonsense knowledge refers to an extensive array of “everyday wisdom”, which is considered to be universally known to humans. For instance, in the sentence, “He hit the window with a bat,” it is obvious to a human reader that the “bat” refers to a sporting equipment rather than the animal. However, for machines, leveraging such knowledge poses considerable challenges, primarily because it is rarely explicitly stated in human communication, which comes in addition to its vast scope and diversity. By combining NLP systems with this “everyday wisdom”, we can significantly enhance their capabilities, thus leading to smoother and more human-like interactions.

In this paper, we aim to enhance the integration of a particularly nuanced form of commonsense knowledge: script knowledge. The term “script” refers to typical sequences of activities that describe well-known situations. To illustrate, consider a “someone opening the door” scenario: an individual reaches for his key, and, if it is not found, he rings the doorbell with the hope that someone inside will open the door. This narrative seems intuitive to a human; so, when given the sentence “He must ring the doorbell”, it is a logical inference that “he forgot his key”. However, a machine lacking this knowledge would struggle to form this connection. Improving script knowledge integration would enable NLP systems to produce better responses to everyday human activities, thus enhancing their practical applicability.

We investigated techniques to augment the script knowledge understanding of an established baseline system known as the TriAN [2]. Our experiments leveraged the MCScript [3] dataset, which is a machine question-answering (QA) dataset that is specifically designed to assess a model’s script knowledge comprehension. We delved deeper into the dataset and the baseline method in subsequent sections. Our main contributions are the following:

• We trained a generative LSTM language model on “scripts” of everyday human activity and used it as a feature encoder.
• We used multiple layers of attention interactions to enable multi-hop reasoning.
• We added positional embeddings to the intermediate features to enhance the temporal sequential reasoning.

2. Background and Related Work

2.1. Machine Reading and Question-Answering Datasets

How can we measure the effectiveness of an NLP system in comprehending a text? Machine reading and question-answering (QA) tasks have become a popular benchmark. Here, a passage of text is provided to the NLP system, which then must answer a series of questions about the text and often given multiple choices for the answer. The accuracy of the system’s responses reflects its text comprehension and reasoning capabilities. A prime example is the Stanford Question Answering Dataset (SQuAD) [4]. It includes ∼23 k paragraphs that have been excerpted from top-ranking Wikipedia articles, with over 100 k associated question–answer pairs that span a broad range of topics. The SQuAD data was collected through crowdsourcing. More recent and larger datasets have also been released, with some applying search engines to partially automate the data collection and increase the scale. For instance, the SearchQA dataset [5] contains over 6.9 million snippets for its 140 k questions, while MS MARCO [6] boasts of over 1 million questions and 8.8 million passages.

In addition to the generic ones, there exist machine QA datasets that are designed to assess specific aspects of performance. Our work is centered on enhancing reasoning based on script knowledge, so we used the MCScript [3] dataset as our benchmark. The MCScript, which is a machine QA dataset that has been specifically engineered to evaluate an NLP system’s comprehension of script knowledge, comprises ∼140 k questions and 2119 narrative texts that depict 110 everyday activity scenarios. The data was gathered via crowdsourcing. Notably, ∼27% of the questions cannot be answered directly from the provided text, thus necessitating the application of script knowledge about the scenarios. This sets the MCScript apart from counterparts such as the SQuAD, which mandates the answers to be present within the given text, thereby potentially allowing the system to generate an answer by simply scanning the text without deeper reasoning. The MCScript dataset has served as the foundation for a script knowledge benchmark [7] at SemEval 2018. An example passage, along with its questions and answer choices, is shown in Figure 1.

2.2. Integrating Commonsense Knowledge

2.2.1. Using Knowledge Graphs

Knowledge graphs are broadly used [8,9,10,11] as a source of commonsense knowledge. These large graphs consist of nodes representing concepts and edges describing the relationships between them. A representative example is ConceptNet [12], which is a freely available, multi-lingual semantic network. ConceptNet comprises triplets of (subject, relation, object), e.g., (“apple”, “IsA”, “fruit”), (“plane”, “CapableOf”, “fly”), and (“car”, “Causes”, “pollution”). The subject and object are multi-word phrases, and the relation is one of the relation types. ConceptNet contains ∼630 k of such triplets and 39 types of relationships. An overview is shown in Figure 2.

ConceptNet is one of the most extensively used commonsense knowledge graphs. For example, ref. [8] generated knowledge embeddings from ConceptNet to enhance the context features for cloze-style QA tasks. Ref. [9] used ConceptNet data to train a network that predicts the relevance score of concepts, thereby assessing the relevance of a candidate answer to a given question in context. Ref. [11] devised two auxiliary training tasks to improve machine reading comprehension, thereby predicting the existence and type of relationships between concepts in provided texts. The ground truth for these sub-tasks has been sourced from ConceptNet.

We also applied ConceptNet in our experiments to acquire the relationship information of concepts, thereby serving as an input for our model. Larger and richer knowledge graphs may bring further improvements, e.g., the Google Knowledge Graph, which is a proprietary knowledge base developed by Google to enhance its search results and other services. It can connect more complex information to answer questions, such as “Who is the president of the country where the White House is located”. Our method is fully compatible with more advanced knowledge graphs, which we leave for future work.

2.2.2. Using Additional Text Corpus

The extra text corpus is also a valuable source of commonsense knowledge. For example, ref. [13] mined object semantic knowledge from Wikipedia articles to generate more plausible text completions. Ref. [14] drew prior statistics from the ProPara [15] dataset (a text corpus focused on scientific procedures, such as photosynthesis) to improve the model’s understanding of entity state changes in scientific processes. In addition, ref. [16] created a new dataset named Common Sense Explanations (CoS-E), which resembles a QA dataset but accompanies each question with a sentence of explanation for its correct answer. A language model (LM) is trained to generate an explanation when given a question–answer pair, with the produced explanation then offered as additional input to the downstream classification model.

Our approach also employs additional text data to acquire commonsense knowledge, especially script knowledge. We trained a generative LM with a story dataset, which we then utilized as a script-knowledge-aware feature encoder. The text in the story dataset combines the ∼2100 passages from the MCScript and an additional 500 passages from the MCTest [17] dataset, both being narrative texts, which contains typical sequences of daily activities, a.k.a script knowledge. Compared to the aforementioned related works, our solution is straightforward and can be trained end-to-end.

3. Method

3.1. Baseline Model

The baseline model we re-implemented is called Three-Way Attentive Networks (TriANs) [2]. It models the interactions between the passage and the question, the question and the answer, and the passage and the answer by using a three-way attention mechanism. An overview of the model framework is shown in Figure 3. It consists of an input layer, an attention layer, and an output layer.

Input Layer: A training example consists of a passage ${\left\{P_i\right\}}_{i=1}^{\left|P\right|}$, a question ${\left\{Q_i\right\}}_{i=1}^{\left|Q\right|}$, an answer ${\left\{A_i\right\}}_{i=1}^{\left|A\right|}$, and a label $y^{\ast }\in \{0,1\}$ as input. $P_i$ is the representation of the i-th word in the passage, which is the same for the question and the answer. For the representation, we used the GloVe word vector [18] for the passage, the question, and the answer (${\mathbf{E}}_{P_i}^{GLOVE}$, ${\mathbf{E}}_{Q_i}^{GLOVE}$, and ${\mathbf{E}}_{A_i}^{GLOVE}$), as well as used the following features:

• POS Embeddings (${\mathbf{E}}_{P_i}^{POS}$ and ${\mathbf{E}}_{Q_i}^{POS}$): Randomly initialized 12-dimensional vectors trained to encode pre-labeled part-of-speech tags (whether the word is a noun, verb, etc.) for passage and question texts.
• NER Embeddings (${\mathbf{E}}_{P_i}^{NER}$): Randomly initialized 8-dimensional vectors trained to encode pre-labeled name–entity recognition tags (whether the word belongs to categories such as people, company, date, etc) for the passage texts.
• REL Embeddings (${\mathbf{E}}_{P_i}^{REL}$): Randomly initialized 10-dimenstional vectors trained to encode a relationship between a word in the passage and any word in the question/answer. Such relationship comes from querying ConceptNet for an edge between the words. If multiple relationships exist, a random one is chosen.
• Handcrafted Features: Handcrafted features include logarithmic term frequency (${\mathbf{E}}_{P_i}^{TF}$) and co-occurrence features (${\mathbf{E}}_{P_i}^{CO}$). Logarithmic term frequency features come from English Wikipedia. Co-occurence features are binary, thus being true if a passage word is found in the question/answer.

$${\mathbf{W}}_{P_i}=[{\mathbf{E}}_{P_i}^{GLOVE};{\mathbf{E}}_{P_i}^{POS};{\mathbf{E}}_{P_i}^{NER};{\mathbf{E}}_{P_i}^{REL};{\mathbf{E}}_{P_i}^{CO};{\mathbf{E}}_{P_i}^{TF}]$$

(1)

$${\mathbf{W}}_{Q_i}=[{\mathbf{E}}_{Q_i}^{GLOVE};{\mathbf{E}}_{Q_i}^{POS}]$$

(2)

$${\mathbf{W}}_{A_i}=[{\mathbf{E}}_{A_i}^{GLOVE}]$$

(3)

Attention Layer: We used word-level attention to model interactions between the passage, the question, and the answer. The model first calculates context vectors for the passage by attending to the question. Then, it calculates context vectors for the answer by attending to the question and the passage. The attention function is represented in (4) and (5). f is a non-linear activation function and is set to $ReLU$. From the attention function, we can compute question-aware passage representations ${\mathbf{W}}_{P_i}^q$, passage-aware answer representations ${\mathbf{W}}_{A_i}^p$, and question-aware answer representations ${\mathbf{W}}_{A_i}^q$ in (6)–(8).

$$At{t}_{seq}(\mathbf{u},{\left\{{\mathbf{v}}_i\right\}}_{i=1}^n)=\sum _{i=1}^n{\alpha }_i{\mathbf{v}}_i$$

(4)

$${\alpha }_i=softma{x}_i(f{({\mathbf{W}}_1\mathbf{u})}^{T}f({\mathbf{W}}_1{\mathbf{v}}_i))$$

(5)

$${\mathbf{W}}_{P_i}^q=At{t}_{seq}({\mathbf{W}}_{P_i},{\left\{{\mathbf{W}}_{Q_i}\right\}}_{i=1}^{\left|Q\right|})$$

(6)

$${\mathbf{W}}_{A_i}^p=At{t}_{seq}({\mathbf{W}}_{A_i},{\left\{{\mathbf{W}}_{P_i}\right\}}_{i=1}^{\left|P\right|})$$

(7)

$${\mathbf{W}}_{A_i}^q=At{t}_{seq}({\mathbf{W}}_{A_i},{\left\{{\mathbf{W}}_{Q_i}\right\}}_{i=1}^{\left|Q\right|})$$

(8)

After that process, three groups of context embeddings are appended to the original embeddings to form the final embeddings. Then, three BiLSTMs are applied to the concatenated embeddings to model the temporal dependency, as shown in (9)–(11).

$${\mathbf{h}}^p=BiLSTM({[{\mathbf{W}}_{P_i};{\mathbf{W}}_{P_i}^q]}_{i=1}^{\left|P\right|})$$

(9)

$${\mathbf{h}}^q=BiLSTM({[{\mathbf{W}}_{Q_i}]}_{i=1}^{\left|Q\right|})$$

(10)

$${\mathbf{h}}^a=BiLSTM({[{\mathbf{W}}_{A_i};{\mathbf{W}}_{A_i}^p;{\mathbf{W}}_{A_i}^q]}_{i=1}^{\left|A\right|})$$

(11)

Output Layer: The question sequence and answer sequence representation—${\mathbf{h}}^q$ and ${\mathbf{h}}^a$, respectively, are summarized into fixed-length vectors $\mathbf{q}$ and $\mathbf{a}$ with self-attention. The self-attention function is defined as in (12) and (13). To represent the passage, we used the sequence attention function defined before to summarize out the passage representation $\mathbf{p}$ by attending $\mathbf{q}$ to ${\mathbf{h}}^p$. These operations are shown in (14)–(16).

$$At{t}_{self}({\left\{{\mathbf{u}}_i\right\}}_{i=1}^n)=\sum _{i=1}^n{\alpha }_i{\mathbf{u}}_i$$

(12)

$${\alpha }_i=softma{x}_i({\mathbf{W}}_2^T{\mathbf{u}}_i)$$

(13)

$$\mathbf{q}=At{t}_{self}({\left\{{\mathbf{h}}_i^q\right\}}_{i=1}^{\left|Q\right|})$$

(14)

$$\mathbf{a}=At{t}_{self}({\left\{{\mathbf{h}}_i^a\right\}}_{i=1}^{\left|A\right|})$$

(15)

$$\mathbf{p}=At{t}_{seq}(\mathbf{q},{\left\{{\mathbf{h}}_i^p\right\}}_{i=1}^{\left|P\right|})$$

(16)

Then, the question and answer vectors are summarized to single vectors using self attention. The passage vectors are summarized to a single vector via bi-linear attention on the question vectors. Finally, we used a bi-linear function to summarize the 3 sequence representations and applied a sigmoid activation function to obtain the final probability score on whether the choice was the correct answer for the question with respect to the passage, which is shown in (17).

$$y=\sigma ({\mathbf{p}}^T{\mathbf{W}}_3\mathbf{a}+{\mathbf{q}}^T{\mathbf{W}}_4\mathbf{a})$$

(17)

Figure 3. Baseline model framework [2].

Figure 3. Baseline model framework [2].

3.2. Error Analysis

In order to improve upon the TriANs, it is important to understand its limitations. In this section, we performed error analysis on the original TriANs model.

3.2.1. Qualitative Analysis

We inspected all the errors made by machine and grouped common problems into the following categories. The main problem clusters are the following:

• Failure to build temporal sequence: From a human perspective, temporal sequences often imply causal relationships. For instance, if event A happens before event B, it is possible that A caused B. This implication becomes stronger if A occurs immediately before B, thereby suggesting that A is the direct cause of B. However, the model frequently identified only indirect causes, thus indicating its struggle to interpret these direct temporal–causal links accurately. This can be mitigated via the better integration of script knowledge.
• Failure to choose based on elimination: This term refers to instances where the incorrect options are explicitly contradicted by evidence in the passage, even though there is no clear evidence supporting the correct choice. In these cases, it might be challenging to identify the correct answer directly. However, the correct response should still have been deduced by a process of elimination.
• Lack of commonsense knowledge: Despite the model leveraging ConceptNet to incorporate commonsense knowledge, it appears this approach is not sufficient. The model could still make non-sensical mistakes.
• Lack of quantification ability: Sometimes to answer the question, it is required to perform some simple computation or estimation on the value. The model was observed to struggle with these operations.
• Weakness in co-reference resolution: The system struggled with a co-reference resolution when the reference was ambiguous. Resolving these ambiguities often requires simple reasoning, which is grounded in both the context and commonsense knowledge. This can also be improved by better commonsense integration.
• Ground truth answers may be wrong: Some ground truth answers seemed to be wrong from the human perspective, which contributes to a small portion of the errors.

3.2.2. Quantitative Analysis

Besides qualitative analysis, we also performed quantitative analysis. In Figure 4, we plotted the distribution of confidence of the model on the choice in 4 cases, as explained in the caption of the figure. From the “pred_t_in_t” and “pred_f_in_f” sub-plots, we can see that the model was usually confident with its correct predictions. In the other two sub-plots, where the model made incorrect predictions, the confidence scores were much lower. This lack of confidence indicates that there was no sufficient information for the model to make confident choices. The addition of more background knowledge may resolve some ambiguity and provide improvements.

We then performed some analysis on the distribution of errors for 10 question scenarios, as shown before in Figure 5. We calculated the error rate of the model for 10 scenarios in Figure 6. From the figure, the “when” category resulted in being the worst. This indicates that the current model struggles with reasoning about temporal sequences, thus suggesting room for improvement through the integration of script knowledge.

3.3. Proposed Method

3.3.1. Methodology Breakdown

To enhance the script knowledge integration for the baseline model, we proposed several techniques that are explained below.

A:

Integrating a Pre-Trained Language Model

We pre-trained a generative language model (LM) with a dataset of narrative passages. The LM was an LSTM that predicts the next word when given the predicted words in an auto-regressive manner. The dataset was made by combining the passages from the MCScripts and MCTest [17], which totals ∼2600 passages. This forms an extended script knowledge base that mitigates overfitting. We then used the pre-trained LM to produce additional feature embeddings for the input text, and we fine-tuned it jointly with the entire model. The auto-regressive generation task naturally made the model more aware of “what happens next” in a series of events, which embodies the script knowledge.

B:

Multi-hop Attention

The attention mechanism has been proven to be an effective method to help the model focus on the most relevant information [19,20,21,22]. It serves this purpose in the baseline method by enabling the model to give more weights to the most helpful parts of the passage for answering the question. However, single-hop attention is insufficient to perform more complex hierarchical reasoning [23,24]. Therefore, we proposed multi-hop attention. Take another look at the example text in Figure 1. For the first question, the model should first attend to heating the food in the passage, which is warm it up. Then, the model should scan up or down to find where the food is heated, which is on the stove top. This two-step process is an example of multi-hop reasoning. The first hop is to locate the key word; the second hop is to locate information before or after the key word, which is closely related depending on the question. Generally, the more indirect the relation between the questioned item and the true answer, the more hops of attention are needed.

C:

Positional Embedding

The attention operation used in the baseline method is invariant to the order of words. However, temporal order is an important aspect of script knowledge. To explicitly represent such sequential information in the process of multi-hop attention, we added positional embedding for each word, which aimed to enhance the model’s reasoning regarding event orderings.

3.3.2. Model Architecture

The overall architecture of our model is shown in Figure 7. The framework can be divided into the following steps.

Step 1:

The First Hop

The first hop is the same as with the attention applied before the BiLSTMs in the baseline model. We denote the input of the first hop as ${\mathbf{W}}_{P_{i1}}$, ${\mathbf{W}}_{Q_{i1}}$, and ${\mathbf{W}}_{A_{i1}}$, in Equations (18)–(20), respectively. Using the same attention function with that in the baseline model, we obtain ${\mathbf{W}}_{P_{i1}}^q$, ${\mathbf{W}}_{A_{i1}}^p$, and ${\mathbf{W}}_{A_{i1}}^q$, respectively.

$${\mathbf{W}}_{P_{i1}}=[{\mathbf{E}}_{P_i}^{GLOVE}]$$

(18)

$${\mathbf{W}}_{Q_{i1}}=[{\mathbf{E}}_{Q_i}^{GLOVE}]$$

(19)

$${\mathbf{W}}_{A_{i1}}=[{\mathbf{E}}_{A_i}^{GLOVE}]$$

(20)

Step 2:

The Second Hop

The input of the second hop is a concatenation of the input of the first hop, the output embeddings from first hop, and the positional embeddings, as shown in Equations (21)–(23). Then, we applied the same attention mechanism as in the first hop and obtained ${\mathbf{W}}_{P_{i2}}^q$, ${\mathbf{W}}_{A_{i2}}^p$, and ${\mathbf{W}}_{A_{i2}}^q$, respectively.

$${\mathbf{W}}_{P_{i2}}=[{\mathbf{W}}_{P_{i1}};{\mathbf{W}}_{P_{i1}}^q;{\mathbf{E}}_{P_i}^{PE}]$$

(21)

$${\mathbf{W}}_{Q_{i2}}=[{\mathbf{W}}_{Q_{i1}};{\mathbf{E}}_{Q_i}^{PE}]$$

(22)

$${\mathbf{W}}_{A_{i2}}=[{\mathbf{W}}_{A_{i1}};{\mathbf{W}}_{A_{i1}}^p;{\mathbf{W}}_{A_{i1}}^q;{\mathbf{E}}_{A_i}^{PE}]$$

(23)

Step 3:

The Output Layer

In the third step, we concatenated the output embeddings from the previous step with additional features, including the embeddings generated from the pre-trained language model, as shown in Equations (24)–(26). ${\mathbf{W}}_{P_{i3}}$, ${\mathbf{W}}_{Q_{i3}}$, and ${\mathbf{W}}_{A_{i3}}$ are the corresponding inputs of BiLSTMs.

$$\begin{array}{c}\hfill {\mathbf{W}}_{P_{i3}}=[{\mathbf{W}}_{P_{i2}};{\mathbf{W}}_{P_{i2}}^q;{\mathbf{E}}_{P_i}^{LMF};{\mathbf{E}}_{P_i}^{POS};\\ \hfill {\mathbf{E}}_{P_i}^{NER};{\mathbf{E}}_{P_i}^{REL};{\mathbf{E}}_{P_i}^{CO};{\mathbf{E}}_{P_i}^{TF}]\end{array}$$

(24)

$${\mathbf{W}}_{Q_{i3}}=[{\mathbf{W}}_{Q_{i2}};{\mathbf{E}}_{Q_i}^{POS}]$$

(25)

$${\mathbf{W}}_{A_{i3}}=[{\mathbf{W}}_{A_{i2}};{\mathbf{W}}_{A_{i2}}^p;{\mathbf{W}}_{A_{i2}}^q]$$

(26)

Finally, we acquired the prediction through the same operations as in the baseline.

4. Experiments

4.1. Hyper-Parameters and Training

We followed the same setting of hyper-parameters for the training as was set in [2]. We used GloVe word vectors. We used LSTMs with a hidden size of 96, a dropout rate of 0.4 after the embeddings and LSTM outputs, a clipping gradient of 10, and batch size of 32. We used Adamax as the optimizer and initially set the learning rate to $2\times {10}^{-3}$. We decayed it by 0.5 after 10 and 15 epochs.

For the language model pre-training, we used a one-layer LSTM with a hidden size of 256. The maximum sequence length for training was set to 60 words. We used a variational dropout [25,26] to prevent overfitting. Our setup is a simple proof-of-concept; more advanced LM architecture implementation and larger text corpus use are fully compatible.

4.2. Experimental Results

4.2.1. Language Model Samples

We demonsrate a few sample text generated by the LM in Figure 8. Our language model was able to generate plausible consecutive events when given an initial sentence, thereby suggesting that it is able to learn and carry script knowledge.

4.2.2. Question-Answering Accuracy

As the dataset we were using was from the SemEval contest, the test set did not come with ground truth labels. Therefore, we evaluated our system against the validation set. As shown in

Table 1. Validation accuracy of model variants for the MCScript dataset. We ablated the effect of handcrafted features (Handcrafted Feat.), LM features (LM Feat.), multi-hop attention (Multi-hop), and positional embeddings (PE). We repeated the experiments for 3 times and reported the average and standard deviation numbers.

Variant	Handcrafted Feat.	LM Feat.	Multi-Hop	PE	Accuracy
TriAN (baseline)	✓				83.42% ± 0.03%
Variant 1					82.42% ± 0.04%
Variant 2	✓	✓			83.49% ± 0.04%
Variant 3		✓			84.05% ± 0.06%
Variant 4		✓	✓		84.13% ± 0.05%
Ours		✓	✓	✓	84.22% ± 0.04%

, our proposed method improved upon the TriANs with respect to the validation accuracy. The best model, which combined all of the three proposed techniques, achieved an 84.22% accuracy, which was 0.8% higher than the baseline model. Upon comparing variants 1, 3, and 4, as well as the proposed model, we notice that all of the introduced techniques contributed to the performance gain. We also studied the interaction between the LM feature and the handcrafted features by comparing the baseline, variant 1, variant 2, and variant 3. While both helped the performance, using the LM features alone seems better than combining the two.

5. Discussion

5.1. Results Analysis

We conducted an ablation study on the components of our method. Comparing the baseline with variant 2 shows that adding the LM features afforded a +$0.07\%$ increase in accuracy. Completely replacing the handcrafted features with the LM features (variant 2 vs. 3) performed even better, thus showing a +$0.63\%$ increase in accuracy over the baseline. This shows that data-driven features that are learned from a large text corpus are a superior replacement for the handcrafted ones such as word frequency. Variant 3 vs. 4 showed that multi-hop attention could afford a +$0.08\%$ increase in accuracy, and adding positional embeddings (variant 4 vs. the proposed method) further afforded a +$0.09\%$ increase in accuracy. The most significant gain came from integrating the LM. Our interpretation is that training this LM on a corpus of narrative text in an auto-regressive manner effectively makes it summarize a script knowledge base, which, in return, enables our model to better comprehend the event sequences. The improvements from multi-hop attention show its effectiveness at reasoning indirect connections between the concepts. The further gains from the positional embeddings prove that it is helpful to make word embeddings that are aware of their positions in the passage, which enhances the model’s awareness of temporal sequences and causal relationships.

5.2. Future Work

Our framework is compatible with many potential improvements. We list a few here as the future work.

A language model with a higher capacity: Extending upon our LM integration, we can obtain pre-trained BERT [27] embeddings and fine-tune the embeddings to our dataset. As BERT is a more capable model that is trained on a much larger text corpus; the embeddings will implicitly carry a large body of commonsense knowledge, including script knowledge, which are expected to perform better than our current language model features.

A better representation for events: In this study, we encoded the input texts in a per-word manner. A potential alternative is memory networks [28,29] that encode each event at the sentence level. With memory networks, each sentence will have a single embedding so that the attention could be applied at the event level rather than at the word level. That may allow the model to focus more on semantics rather than syntax. Similarly, we would also like to explore training the language model at the sentence level by using the hidden representations to predict which sentence should directly follow the previous sentence.

A stronger knowledge graph: In this study, our REL feature encoded the limited 39 types of relationships from the ConceptNet. It was also bounded by the entities that exist in the ConceptNet. Our framework is compatible with larger knowledge graphs that contain more complex relationships, e.g., the Google Knowledge Graph.

6. Conclusions

In this paper, we presented methods to improve script knowledge integration and reasoning, which included the following: 1. Integrating a generative language model that learns script knowledge from a large text corpus. 2. Using multi-hop attention to support multi-hop reasoning. 3. Using positional embeddings to enhance the reasoning about event sequences and causal relationships. The experiments on the MCScript dataset demonstrate the effectiveness of our framework.