Multi-Hop Knowledge Graph Question Answer Method Based on Relation Knowledge Enhancement
Round 1
Reviewer 1 Report
The authors present a multi-hop knowledge graph question-answer model based on relation knowledge enhancement. The approach comprises different layers: knowledge aggregation, encoding, interaction, and link reasoning. The model mentioned was tested using MetaQA, WebQSP, and ComWebQ datasets. The authors performed different experiments utilizing the Hits@1 metric on the mentioned datasets.
The multi-hop problem on knowledge graph question answering is an interesting research topic. In this sense, the submitted paper is a valuable contribution. However, the article must be prepared for journal publication. The following are my major points of criticism:
Introduction
- From a general viewpoint, the introduction needs to be improved.
- The authors focused on GraftNet, PullNet, and TransferNet models without context. It may be mentioned, but these models are part of the related work section.
- An additional effort needs to be performed to clarify the problem of this contribution.
Models and methods
- The third contribution and how it is achieved in the article need to be clarified.
- In the knowledge aggregation layer, what does relevant knowledge mean? How is relevant defined and identified?
- Are the authors losing knowledge in their model? For instance, Figure 4 seems that “Tesla CEO Musk” is not considered because the authors only consider the text's initial and final elements.
- With respect to the Encoding layer, what happens with diverse text elements? The authors mentioned tokenization, but why were not NLP techniques (e.g., Name Entity Recognition, lemmatization, or stop words) utilized? The lack of these techniques, for instance, entails that different tokens are repeated (found, founded, founding).
Experiment
- Are data of WebQSP dataset included in ComWebQ?
- Did the authors perform some fine-tuning with the mentioned hyperparameters?
- How was the threshold defined? Why is it 0.7?
- “Model is optimized using RAdam.” An additional explanation about this optimization is required.
- Why was the Hits@1 metric utilized? Did the authors test with additional Hits@X?
- Checking the results, the authors’ approach did not offer better results than existing approaches for MetaQA. Why does it happen? Why are the existing approaches better?
- On the other hand, the authors’ approach seems to work better when text data sources are considered. Why does it happen?
- “For the incomplete KG case where only 50% of the original triples are present,” What is it mean? More details are needed to understand this evaluation section.
- The authors mentioned “sufficient information.” Which is the minimal amount of information required by their approach?
Conclusions
- This section needs to improve.
Minor issues:
- Multiple initials appear along the article without a complete name.
- What are the differences between MetaQA 2-hop and MetaQA 2-hop in Table 1?
Author Response
Response to Reviewer 1 Comments
Point 1: Introduction
- From a general viewpoint, the introduction needs to be improved.
- The authors focused on GraftNet, PullNet, and TransferNet models without context. It may be mentioned, but these models are part of the related work section.
- An additional effort needs to be performed to clarify the problem of this contribution.
Response 1: Thank you for your kind comments on our article , we feel very sorry that we did not provide enough information about our contribution.
-According to your suggestions, we have modified the structure of the introduction in chapter 1;
-We mention the examples of GraftNet, PullNet, and TransferNet models only to explain that recent studies on multi-hop KGQA have tried to use external texts to deal with the sparsity of KG;
-We adapt the introduction to clarify our contribution in chapter 1. (in red)
Point 2: Models and methods
- The third contribution and how it is achieved in the article need to be clarified.
- In the knowledge aggregation layer, what does relevant knowledge mean? How is relevant defined and identified?
- Are the authors losing knowledge in their model? For instance, Figure 4 seems that “Tesla CEO Musk” is not considered because the authors only consider the text's initial and final elements.
- With respect to the Encoding layer, what happens with diverse text elements? The authors mentioned tokenization, but why were not NLP techniques (e.g., Name Entity Recognition, lemmatization, or stop words) utilized? The lack of these techniques, for instance, entails that different tokens are repeated (found, founded, founding).
Response 2: Thank you for your nice comments on our article, we feel very sorry that we did not provide enough information about our idea.
-The third contribution we clarify in chapter 1.
-In the knowledge aggregation layer, the relevant knowledge is the text knowledge about the topic entity recognized in the question.
-We feel very sorry that we did not provide enough information about the encoding layer, we describe the encoding layer with more detail in chapter 3.2.(in red)
Point 3: Experiment
- Are data of WebQSP dataset included in ComWebQ?
- Did the authors perform some fine-tuning with the mentioned hyperparameters?
- How was the threshold defined? Why is it 0.7?
- “Model is optimized using RAdam.” An additional explanation about this optimization is required.
- Why was the Hits@1 metric utilized? Did the authors test with additional Hits@X?
- Checking the results, the authors’ approach did not offer better results than existing approaches for MetaQA. Why does it happen? Why are the existing approaches better?
- On the other hand, the authors’ approach seems to work better when text data sources are considered. Why does it happen?
- “For the incomplete KG case where only 50% of the original triples are present,” What is it mean? More details are needed to understand this evaluation section.
- The authors mentioned “sufficient information.” Which is the minimal amount of information required by their approach?
Conclusions
- This section needs to improve.
Response 3: Thank you for your nice comments on our article, we feel very sorry that we did not provide enough information about our experiment.
-CompWebQ is an extended version of WebQSP by extending the question entities or adding constraints to answers, while they are different. CompWebQ forms patterned complex questions based on the expansion of SPARQL statements from the WebQSP dataset, and then manually paraphrases the complex questions to form natural language questions.
-Yes, we perform some fine-tuning with the hyperparameters.
-The response paths with score greater than the threshold are highlighted, while the paths with score less than the threshold are ignored. In this way, the operating range will significantly reduce. The choice of the threshold according to experiments. We choose 0.7 based on the accuracy and the processing speed. If the threshold >0.7, some correct links may be pruned; If the threshold<0.7, the time consumption is huge. The value of the threshold can float slightly.
-we add an explanation about optimizer RAdam in chapter 4.3.
-Metric hit@1 is a standard assessment that measures the ratio across all validation, i.e. the entity with the highest score belongs to the correct answer. If the QA system provides a single entity and that entity is correct, then we treat the right prediction as the correct one. This evaluating indicator is popular and publicly recognized. We compare the metric with other models.
-The performance of our model in the single-hop problem is similar to SOTA, although it is not the best result, the gap is not significant. In the case of multi-hop problems and lack of KB, our model achieves better results because the strength of our model lies in dealing with multi-hop complex problems. We clarify our idea in 4.4.
-We propose a tree structure to fuse relation structures in label form and text form to achieve relation knowledge enhancement, and we believe that relational augmentation allows semantic association across hops, and performs especially well on KGs with missing relations, as augmenting relational representations by fusing external textual knowledge with global attention is effective.
-”50% of the KG” is that we randomly delete half of the node information of the KG.
-The “sufficient information” is meaning that the answer cannot reason with the KG, while can reason in the text information.
According to your suggestions, we have supplemented several data here and corrected several mistakes in our previous draft about the experiment section. (in red)
Point 4: Minor issues:
- Multiple initials appear along the article without a complete name.
- What are the differences between MetaQA 2-hop and MetaQA 2-hop in Table 1?
Response 2: We feel very sorry that our careless mistakes. Thank you for your reminding.
-we change the initials problem( in red).
-We feel very sorry that our careless mistakes, and we change the “MetaQA 2-hop” to “MetaQA 3-hop” in the Table 1 .
Author Response File: 
Author Response.pdf
Reviewer 2 Report
The paper presents enhancements to existing knowledge graph algorithms to improve question answering. The research is complex with a lot of components which means the explanation of how the overall approach works is sometimes vague or unclear. Readers who are not intimately familiar with some of the background approaches will struggle to understand how this system works, and that could devalue the impact of the paper. In my comments below, I note a few places where the authors should add some detail. I think it would also be helpful for them to provide a specific example showing the system reasoning from question through answer, showing a few generated graphs and trees as the system processes the question. But that might take up too much space.
My comments below mostly are grammatical or related issues, but there are some comments that also indicate where the authors might want to provide more detail or description.
Overall, this is a good paper and should be published but not before at least one more revision to clean up some of the mistakes I note below.
Line 25-26: Should read "Intelligent question-answering plays ... It requires that machines understand free-form..."
Line 28: spell out SQTA, explain what a hop is before mentioning "a single hop"
Line 33: "attended to handle" is not the correct phrase, maybe you mean "attempted"
Lines 34-35: Readers may not understand "sequential relation paths" or what the hidden features are you reference, be more specific
Line 36: "aimed at combining" not "aimed to combine"
Line 44: again, vague, what are the "topic entities"?
Lines 45-46: spell out GCN, again this is somewhat vague, what is k and who specifies the threshold, and what is the threshold compared to? That is, how are node values computed to compare against the threshold?
Lines 46-49: this is a very long, run-on sentence, try to break it into shorter sentences.
Line 51: your explanation for weakness is unclear, waht are "interpretable inference paths" and why should too few of them be a problem?
Line 56: what does noisy mean in this context
Line 69: should probably be "manually defined, constrained predicates" (add a ,). You have a stray period after "Text form"
Somewhere around this paragraph (lines 67-77) you should probably insert a figure to illustrate a KG.
Line 78: why is a tree structure innovative? trees are used throughout AI, I'd drop the word "innovatively"
Line 82: "propose" to match the other two contributions
Section 2 opening paragraph: very long, try to break it into smaller paragraphs, perhaps one per reference.
Line 103: spell out INR.
Line 108: spell out VRN
Line 117: spell out SRN
Second paragraph: again, break this into smaller paragraphs, perhaps starting with "Embed KGQA uses..." and another at "He et al" and at "Miller et al"
Line 128: what is Roberta? you probably need a brief explanation
Lines 138-140: bad grammar, the clause ",which can simultaneously" is unended, I suggest you drop ", which"
Line 177: "call them labeled form and text form, respectively" should add "from left to right"
Line 182: I'd again drop "innovatively". Let others determine if in fact your solution is innovative or not.
Line 186: "sent it to" --> "send it to"
Line 187: unclear what "stitch" means, a more descriptive word would be useful
Line 191: what is the hidden layer of the question?
Line 207: "The relation" --> "A relation"
Line 209: Either "suppose b denotes" should start a new sentence or the comma before it should be a semicolon.
Lines 211-214: ungrammatical sentence, also "denote" should be "denotes", try to rewrite this sentence, perhaps breaking it into two sentences.
Line 232: stray . after "Figure"
Line 250: "As shown in figure5" --> add to the previous sentence as ", as shown in figure 5."
Section 3.3 opening paragraph: explain what global attention means
Line 293: Don't start a sentence with "Where", I suggest you remove the period from the end of equation (3)
Line 201: inconsistency in capitalizing section headers, should probably be "Link Reasoning Layer"
Line 312: same issue
Line 319: this time you are not starting a new sentence ("where") but you are still ending (6) with a period, remove it
Line 331: "And then so on, ..." is not correct. Maybe drop "And then so on," and start with "We can now get"
Line 347: "chose" not "choice", why did you choose the Vanilla version?
Line 349: space after the .
Lines 359, 363, 382: again, inconsistent with your header titles and capitalization
Line 394: "4 hops" (or better yet, "four hops"). You would say "4-hop" like you do for 1-hop, but when saying "hops" make it "four hops"
Line 397: is this a significant improvement over TransferNet or PullNet? It looks like a modest improvement over those for ComWebQ and a modest improvement over TransferNet for WebQSP
Table 4: You have a footnote (starting on line 402) but I didn't see where you place the superscript 1 in the text or figure
Line 413: "We conducted", and after dataset, should be a , not a . ("As shown in figure 8." is not a sentence)
Lines 439-440: ungrammatical, this is a clause, perhaps that should be part of the previous sentence
Author Response
Response to Reviewer 2 Comments
Point 1: Line 25-26: Should read "Intelligent question-answering plays ... It requires that machines understand free-form..."
Line 28: spell out SQTA, explain what a hop is before mentioning "a single hop"
Line 33: "attended to handle" is not the correct phrase, maybe you mean "attempted"
Lines 34-35: Readers may not understand "sequential relation paths" or what the hidden features are you reference, be more specific
Line 36: "aimed at combining" not "aimed to combine"
Line 44: again, vague, what are the "topic entities"?
Lines 45-46: spell out GCN, again this is somewhat vague, what is k and who specifies the threshold, and what is the threshold compared to? That is, how are node values computed to compare against the threshold?
Lines 46-49: this is a very long, run-on sentence, try to break it into shorter sentences.
Line 51: your explanation for weakness is unclear, what are "interpretable inference paths" and why should too few of them be a problem?
Line 56: what does noisy mean in this context
Line 69: should probably be "manually defined, constrained predicates" (add a ,). You have a stray period after "Text form"
Somewhere around this paragraph (lines 67-77) you should probably insert a figure to illustrate a KG.
Line 78: why is a tree structure innovative? trees are used throughout AI, I'd drop the word "innovatively"
Line 82: "propose" to match the other two contributions
Response 1: Thanks for your suggestions, we have add the details of the evaluation metrics in 4.4 Experiment results and analysis. (in red)
Line 25-26: we change the grammatical errors.
Line 28: we spell out the SOTA, and the single hop is that just need one triple or relation to reason the answer.
Line 33: we change “attended to handle” to “attempted”
Lines 34-35: We describe “sequential relation paths” and “hidden features” in the following example model.
Line 36: we change "aimed to combine" to "aimed at combining" .
Line 44: "topic entities" is the key information and entity recognized from the question, which is important for reasoning the question, for example, “When did the ‘Father of ChatGPT’ serve as CEO?”, the topic entity is “Father of ChatGPT”, and the keyword is ”When”, “CEO”.
Lines 45-46: We spell out the GCN; The score greater than the threshold are highlighted, while the score less than the threshold are ignored. In this way, the operating range will significantly reduce. The choice of the threshold according to experiments.
Lines 46-49: we break the long sentence into short sentences in line .
Line 51:These methods are usually weak in terms of interpretability because neural network models are mostly black-box， and it is difficult to evolve interpretable inference paths.
Line 56:Question parsing is polysemous, and it is nontrivial for the QA system to match those referred entities to the knowledge graph.
Line 69: We are really sorry for our careless, thanking for your reminding. we add a ”,” between “manually defined” and “constrained predicates”. we remove the stray period after "Text form”. We insert a figure 1 to illustrate a KG.
Line 78:We are really sorry for our less rigorous, thanking for your reminding. we drop the word “innovatively”.
Line 82: We are really sorry for our careless, thanking for your reminding. We change the ”propose” in line 91.
Point 2: Section 2 opening paragraph: very long, try to break it into smaller paragraphs, perhaps one per reference.
Line 103: spell out INR.
Line 108: spell out VRN
Line 117: spell out SRN
Second paragraph: again, break this into smaller paragraphs, perhaps starting with "Embed KGQA uses..." and another at "He et al" and at "Miller et al"
Line 128: what is Roberta? you probably need a brief explanation
Lines 138-140: bad grammar, the clause ",which can simultaneously" is unended, I suggest you drop ", which"
Line 177: "call them labeled form and text form, respectively" should add "from left to right"
Line 182: I'd again drop "innovatively". Let others determine if in fact your solution is innovative or not.
Line 186: "sent it to" --> "send it to"
Line 187: unclear what "stitch" means, a more descriptive word would be useful
Line 191: what is the hidden layer of the question?
Line 207: "The relation" --> "A relation"
Line 209: Either "suppose b denotes" should start a new sentence or the comma before it should be a semicolon.
Lines 211-214: ungrammatical sentence, also "denote" should be "denotes", try to rewrite this sentence, perhaps breaking it into two sentences.
Line 232: stray . after "Figure"
Line 250: "As shown in figure5" --> add to the previous sentence as ", as shown in figure 5."
Response 3: We are really sorry for our careless, thanking for your reminding. We have corrected these mistakes based on your suggestions. (in red)
Section2 and second paragraph, we break this into several smaller paragraphs.
Line 103/108/117: we spell out “INR” with “Intesrpretable Reasoning Network”, spell out “VRN” with “Variational Reasoning Network”, spell out “SRN” with “Stepwise Reasoning Network”.
Line 128: Roberta model is a pre-trained model with a larger number of parameters, larger batch size and more training data, we change it in line .
Lines 138-140: We are really sorry for our grammar mistake. We drop out the “which”.
Line 177: we add "from left to right" before "call them labeled form and text form, respectively"
Line 182: We are really sorry for our less rigorous, thanking for your reminding. we drop the word “innovatively”.
Line 186:we change "sent it to" to "send it to".
Line 187: we change the word "stitch" to “concatenate”
Line 191: The hidden layer of the question is the character that we can achieve through the pre-trained model through the question.
Line 207: we change the word "The relation" to "A relation".
Line 209: we change the comma before "suppose b denotes" to a semicolon.
Lines 211-214: we change the word "denote" to be "denotes", and break the sentence into two sentences.
Line 232: we change the mistake of spelling errors.
Line 250: we change the sentence "As shown in figure5" to ", as shown in figure 5."
Point 3: Section 3.3 opening paragraph: explain what global attention means
Line 293: Don't start a sentence with "Where", I suggest you remove the period from the end of equation (3)
Line 201: inconsistency in capitalizing section headers, should probably be "Link Reasoning Layer"
Line 312: same issue
Line 319: this time you are not starting a new sentence ("where") but you are still ending (6) with a period, remove it
Line 331: "And then so on, ..." is not correct. Maybe drop "And then so on," and start with "We can now get"
Line 347: "chose" not "choice", why did you choose the Vanilla version?
Line 349: space after the .
Lines 359, 363, 382: again, inconsistent with your header titles and capitalization
Line 394: "4 hops" (or better yet, "four hops"). You would say "4-hop" like you do for 1-hop, but when saying "hops" make it "four hops"
Line 397: is this a significant improvement over TransferNet or PullNet? It looks like a modest improvement over those for ComWebQ and a modest improvement over TransferNet for WebQSP
Table 4: You have a footnote (starting on line 402) but I didn't see where you place the superscript 1 in the text or figure
Line 413: "We conducted", and after dataset, should be a , not a . ("As shown in figure 8." is not a sentence)
Lines 439-440: ungrammatical, this is a clause, perhaps that should be part of the previous sentence
Response 3: We are really sorry for our careless, thanking for your reminding. We have corrected these mistakes based on your suggestions. (in red)
Section 3.3 :Global attention will compute the weight vector of the subject entity with all contexts, while local attention will only compute the weight vector of the word with the contexts within its window size.
Line 293: We are really sorry for our careless, we change the mistake of spelling errors.
Line 201/312: We are really sorry for our careless, we change the “Link reasoning Layer” to “Link Reasoning Layer”.
Line 319:We are really sorry for our careless, we change the mistake of spelling errors.
Line 331: We drop the word “And then so on”.
Line 347: We change "choice" to "chose". NTM is a translation of each question in the Vanilla dataset into French and then translates back into English using beam search to obtain a paraphrased question. Audio is the voice version.
Line 349: we add the space after the ” .”
Line 394: we change the "4 hops" to "four hops".
Table 4: we change the footnote mistake.
Line 413: we change the ” .” to “,” .
Lines 439-440: We are really sorry for our grammar mistake, we change the sentence.
Author Response File: Author Response.pdf
Reviewer 3 Report
Dear esteemed Editors and Authors,
Thank you for giving me a chance to review this article.
The reviewer hereby submits the review report for the article "Multi-hop Knowledge Graph Question Answer Method Based on Relation Knowledge Enhancement."
I want to express my appreciation for the considerable effort invested in this work and offer constructive suggestions to enhance its quality further.
Thank you very much.
Comments for author File: Comments.pdf
Author Response
Response to Reviewer 3 Comments
Point 1: The title of this article is: "Multi-hop knowledge graph Question Answer
Method based on Relation Knowledge Enhancement." The phrase "knowledge graph Question Answer Method" appears to be a confusing noun string. Consider rewriting the sentence for clarity. Here is the reviewer's suggestion: "Enhancing Question Answering in Knowledge Graphs through Multi-hop Reasoning and Relation Knowledge".
Response 1: We are thanking for your reminding. (in red)
Point 2: Some sentences are pretty technical and may be difficult for readers without a
strong background in the field.
Response 2: We are really sorry for our negligence, thanking for your reminding. We have corrected these mistakes based on your suggestions in chapter 1. (in red)
Point 3: The article proposes an attractive approach for entity-centric question
answering with multi-hop reasoning. The interaction and link reasoning layers
are designed to tackle the specific challenges faced in such tasks. However,
without more context, it isn't easy to evaluate the effectiveness of the proposed
model, and the excerpt only provides a glimpse into the technical details of the
approach. The article would benefit from further explanations and evaluations
of the proposed method
Response 2: Thanking for your reminding. We have corrected these mistakes based on your suggestions in chapter 3. (in red)
Point 4: The conclusion of this paper is well-written and provides a clear and concise
summary of the paper's main contributions and findings. However, it would
have been helpful if the authors had discussed some limitations and potential
future improvements of their proposed model. It would have been beneficial
to briefly discuss the practical implications and applications of their proposed
model in real-world scenarios.
Response 2: Thanking for your advice. We discuss the limitations and potential
future improvements of the model in chapter 5. (in red)
Point 5: This paper lacks adequate referencing to substantiate its claims. The author
should increase the number of citations, which is typically from 35 to 65
Response 2: Thanking for your reminding. We have corrected these mistakes based on your suggestions in Reference. (in red)
Point 6: The authors of this article could benefit from more thorough proofreading and
editing to improve the clarity and readability of the writing
Response 2: We are really sorry for our careless, thanking for your reminding. We perform thorough proofreading and editing to improve the clarity and readability of the writing based on your suggestions. (in red)
Author Response File: Author Response.pdf
Reviewer 4 Report
The research presented in the abstract proposes a new approach for solving the challenging task of multi-hop knowledge graph question answering (KGQA) by leveraging both label and text relations through global attention for relation knowledge augmentation. The authors highlight the problem of knowledge graph (KG) sparsity due to missing links and how their proposed approach overcomes this challenge by using relevant external texts.
The authors provide a clear and concise overview of the research, explaining the motivation for the study, the proposed approach, and the experimental results. However, if the authors add some additional details about the evaluation metrics used to measure the performance of the proposed approach, it would be nice.
Overall, the paper is well-written and informative, and the research presented appears to be promising in addressing the challenges of multi-hop KGQA.
There are some mistypos such as "3. we" -> "3. We" at the page 1.
Thanks.
Author Response
Response to Reviewer 4 Comments
Point 1: However, if the authors add some additional details about the evaluation metrics used to measure the performance of the proposed approach, it would be nice.
Response 1: Thanks for your suggestions, we add the details of the evaluation metrics in chapter 4.4 ,Metric hit@1 is a standard assessment that measures the ratio across all validation, i.e. the entity with the highest score belongs to the correct answer. If the QA system provides a single entity and that entity is correct, then we treat the right prediction as the correct one. This evaluating indicator is popular and publicly recognized. We compare the metric with other models. (in red)
Point 2: There are some mistypos such as "3. we" -> "3. We" at the page 1.
Response 2: We are really sorry for our careless, thanking for your reminding. We correct these mistakes based on your suggestions. (in red)
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The authors included most of my previous comments in the new version of their contribution. In this sense, the article is ready to be published in the journal.
