CRP-RAG: A Retrieval-Augmented Generation Framework for Supporting Complex Logical Reasoning and Knowledge Planning
Round 1
Reviewer 1 Report
Comments and Suggestions for AuthorsThe paper attempts to enhance the existing Retrieval-Augmented Generation (RAG) algorithm by introducing a graph-based logical reasoning framework to address the limitations of RAG in complex reasoning and knowledge utilization. However, there are several issues that warrant further refinement and clarification in the paper:
1. Excessive Length in the Introduction: The introduction section is overly lengthy. There is a significant amount of repetition between the second and third paragraphs, and the fourth paragraph could be more concise. This would make the introduction more succinct and easier to follow. Additionally, the introduction lacks a brief roadmap of the paper, which would help guide the reader through the structure of the article.
2. Unclear Focus of the Research: The core theme of the paper appears somewhat ambiguous. The paper should focus on developing a RAG algorithm that is oriented toward complex logical reasoning and knowledge planning. However, a large portion of the content, particularly in the second and third paragraphs of the introduction, as well as Sections 2.1 to 2.3 in the literature review, primarily discusses the deficiencies of the RAG algorithm. At the same time, there is considerable focus on the issues surrounding large language models (LLMs) in Section 2.4 of the review. This dual focus makes it unclear what the main contribution of the paper is. It seems the paper not only aims to improve RAG but also incorporates elements of LLMs. However, the role of LLMs in addressing the identified issues of RAG remains underexplained. Specifically, what unique contribution do LLMs make in overcoming the problems discussed regarding RAG? The introduction of LLMs shifts the focus of the paper and requires clearer justification and elaboration on how they integrate with and enhance the proposed framework.
3. Over-reliance on Large Language Models: Several components of the proposed algorithm heavily rely on large language models. For instance, Equation (4) uses an LLM to generate reasoning graphs, Equation (8) leverages an LLM to evaluate the comprehensiveness or similarity of two or more knowledge structures, and Equation (9) uses an LLM to assess the knowledge richness of multiple reasoning graphs. These steps depend heavily on the LLM's logical reasoning abilities and knowledge boundaries. While LLMs generally perform well with common-sense knowledge, their reliability in specialized knowledge reasoning and boundary identification remains questionable and deserves further investigation. Relying entirely on LLMs for these tasks could potentially undermine the robustness and reliability of the model. This dependency on LLMs needs to be critically evaluated in the paper, particularly in terms of how it might impact the model's accuracy and trustworthiness, especially when dealing with more domain-specific or technical content.
4. Tips: I didn’t find an introduction of the term “CRP-RAG” in this manuscript.
These points should be addressed to improve the clarity, focus, and rigor of the manuscript.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Comments and Suggestions for AuthorsThis paper proposes the CRP-RAG framework, which leverages reasoning graphs to model complex query reasoning processes with better accuracy. The framework dynamically adjusts reasoning paths based on evaluation results, ensuring knowledge sufficiency for answer generation. The experimental results indicate that CRP-RAG outperforms state-of-the-art LLMs and RAG baselines across three reasoning and question-answering tasks, offering enhanced factual consistency and robustness.
There are some issues:
Although the framework is designed for complex queries, examples or case studies illustrating its handling of highly intricate queries would strengthen the claims.
Dynamic reasoning path adjustment could introduce computational overhead. Have you evaluated the framework’s efficiency, and how does it compare to traditional RAG methods?
It would be beneficial to include an error analysis to identify failure cases and areas where CRP-RAG might underperform.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 3 Report
Comments and Suggestions for AuthorsThe paper needs the following updates:
1. the abstract needs to include a summary of the experiment results and an explanation
2. the contribution of the paper needs more explanation with discussing the impact of these contributions on the field.
3. there is no clear comparison with the literature to clarify the impact of the paper.
4. the framework is good but needs more explanation through different perspectives such as a flowchart, use cases, ...etc.
5. the presented formulas need to be explained according to the value, the terms,... etc.
6. the conclusion lacks most of the required discussion. including discussion for the reached results, relations with the contribution, challenges, and clear future work
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Reviewer 4 Report
Comments and Suggestions for AuthorsThe paper proposed CRP-RAG framework which employs reasoning graphs to model complex query reasoning processes more comprehensively and accurately. This is an interesting area. However, needs minor improvement. kindly see suggestions below for improvement.
Methodology:
Kindly provide details of resources used for the implementation.
Kindly provide more details including the mathematical representation on the evaluation metrics used (EM, F1 and Acc-LM).
Result:
Your results show improvements to existing frameworks. Kindly provide scientific justification to this effect. Also, demonstrate using examples (this can be in a Table or screenshot).
Conclusion:
Kindly add a subsection to discuss the theoretical and practical implications of your research output.
What are the limitations of your framework?
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
Round 2
Reviewer 2 Report
Comments and Suggestions for AuthorsThe revised version has addressed my concerns.
Reviewer 3 Report
Comments and Suggestions for Authorsthe authors addressed all comments
