Research on Automatic Question Answering of Generative Knowledge Graph Based on Pointer Network

Round 1

Reviewer 1 Report

This article suggests a current and attractive topic for the academy. The research is timely and worthwhile. The research problem is clearly defined. The authors provide fresh insight into the field.

The work structure is excellent and well-articulated. The literature review is detailed and thorough. To review the scientific sources from the industry can offer work of his colleagues:  Fedushko, S., Ustyianovych, T., Gregus, M. (2020) Real-time high-load infrastructure transaction status output prediction using operational intelligence and big data technologies. Electronics (Switzerland), Volume 9, Issue 4, Article number 668. DOI: 10.3390/electronics9040668

The results are explained in a clear and detailed manner.

Congratulations on a job well done.

Author Response

Hello, thank you for your recognition of our work. The papers you recommend have also given us a lot of inspiration. Thanks again.

Author Response File: Author Response.pdf

Reviewer 2 Report

Summary. This paper studies the question answering problem based on knowledge graphs. The proposed system contains three modules (1) knowledge vocabulary construction, (2) word vector acquisition, and (3) generative model building. Each module is well designed based on the established techniques of NLP and deep learning, such as BiLSTM-CRF and BERT. In the experiment study, this paper partially shows improvement of accuracy, compared with existing methods, but there is still a room to improve.

Strengths.

S1. The paper formulates the question answering problem properly and states clearly what it does. Given vocabularies, questions, and answers, it first constructs the knowledge graphs based on word frequencies, and then learns a generative question answering system, using BERT.

S2. The proposed system is well designed and each module has selected proper techniques. Specifically, to construct the word list, BiLSTM-CRF is used to identify entities and triples. The generative question answering module adopts the BERT and frequency semantic as features, and learns pointer generator network to generate answers.

S3. The paper showcases Chinese Knowledge Base Question Answering, which is an interesting and a challenging task, given the fact that the semantic of Chinese highly depends on the contexts of a sentence.

Weaknesses.

W1. For word frequency features, the paper simply adopts term frequencies (TF). As we know, the most frequent words are stop words, such as the, a, is, which are almost useless in NLP models and add noises to models. To mitigate this issue, TF-IDF (term frequency-inverse document frequency) may need be considered to replace TF. Otherwise, the author should compare TF-IDF as a baseline.

W2. When there is a missing entity or relationship in the knowledge graph, the proposed method simply adds it to the data, which aims to reduce the Out-Of-Vocabulary problem. However, when the missing entity or relationship is erroneous (e.g., they are extracted from the Web), this may downgrade the data quality of the knowledge graphs and further make the model inaccurate.

W3. The paper has several presentation problems. For example, several places miss citations or have acronyms without clarification. See details.

Details.

D1. On page 1, the last sentence “In order to better meet the needs of this requirement, knowledge graph technology is proposed” needs to cite a paper.

D2. On page 2, in the first paragraph, second last sentence, there are “other English knowledge bases” and “other Chinese knowledge bases”. Both of these two need references, respectively. Otherwise, they should be removed.

D3. In the second paragraph, the sentence "Which is the largest city in France or the capital of the United States?" is confusing, as the expected answer is “Paris”. Hence, “or the capital of the United States” should be removed to keep it simple and clear.

D4. In the last third line of Page 2, I guess “tf algorithm” refers to term frequency. If it occurs the first time in this paper, the full name is needed. Similarly, in the last line of Page 2, PGN should be clarified first.

D5. On Page 3, line 120, “and it has become a focus of attention by domestic and foreign researchers.” I guess the authors want to say deep learning gains more popularity, but separating domestic and foreign researchers looks odd.

D6. On Page 3, line 131, “OOV and word rare problems” need to be explained.

D7. On Page 3, line 139, “Seq2Seq framework” need a citation and its idea need to be explained or summarized briefly.

D8. On Page 4, line 156 to line 157, there are three modules proposed in this framework. However, it is hard for readers to link these modules to Figure 1. Could the authors revise Figure 1 and label each of the modules?

D9. Also, in Figure 1, is the “Answer” box used as input for model learning or knowledge vocabulary construction? The authors may need to add an arrow starting from the “Answer” box.

D10. On Page 5, line 182 to line 183, there are “jieba word segmentation” and “BiLSTM-CRF neural networks” that need citations.

D11. On Page 5, line 192 and 193, BiLSTM and CRF modules need citations, respectively, and need to be explained intuitively, before give the acronyms.

D12. On Page 7, line 246 to line 247, there is a statement “The traditional Seq2Seq model cannot handle OOV words and is repetitive.” Why? The authors need provide explanations intuitively or give a citation that can conclude this.

D13. On Page 8, line 259, “CPT to BERT” need citations.

D14. On Page 16, Figure 6 shows the accuracy of proposed approach becomes lower when the knowledge vocabulary increases. The authors explain this as “some more useless nouns increase, which will also lead to inaccurate answers” in line 456 to line 457, which confirms the W2 mentioned above. In addition, when vocabulary size is around 6000 to 7000, the proposed method does not outperform the three baseline methods. Hence, the effectiveness of the proposed method needs more justification, by either more experiment results over other datasets or theoretically explanations (e.g., perhaps noise terms are added by TF etc.). Otherwise, it cannot claim in the Abstract “The experimental results show that the proposed method can achieve superior performance on public datasets than other methods.”

Author Response

Hello, we have made improvements based on your comments and added some information on the last page of the submitted file, such as the line number of the paper revision. Thank you for your valuable comments.

Author Response File: Author Response.pdf