Co-MedGraphRAG: A Collaborative Large–Small Model Medical Question-Answering Framework Enhanced by Knowledge Graph Reasoning

Abstract

Large language models (LLMs) have demonstrated significant capabilities in natural language processing (NLP), but they often encounter challenges in the medical domain. This can result in insufficient alignment between generated answers and user intent, as well as factual deviations. To address these issues, we propose Co-MedGraphRAG, a novel framework combining knowledge graph reasoning with large–small model collaboration, aimed at improving the structural grounding and interpretability of medical responses. The framework operates through a multi-stage collaborative mechanism to augment question answering. First, a large language model constructs a question-specific knowledge graph (KG) containing pending entities (denoted as “none”) to explicitly define known and unknown variables. Subsequently, a hybrid reasoning strategy is employed to populate the pending entities, thereby completing the question-specific knowledge graph. Finally, this graph serves as critical structured evidence, combined with the original question, to augment the large language model in generating the final answer, implemented using Qwen2.5-7B and GLM4-9B in this paper. To evaluate the generated answers, we introduce a larger-parameter LLM(GPT-4o) to assess performance across five dimensions and compute an overall score. Experiments on three medical datasets demonstrate that Co-MedGraphRAG achieves consistent improvements in relevance, practicality, and structured knowledge support compared with mainstream Retrieval-Augmented Generation (RAG) frameworks. This work serves as a reference for researchers and developers designing medical question-answering frameworks and exploring decision-support applications.

1. Introduction

The rapid development of large language models (LLMs) has greatly advanced research in natural language processing, including applications in intelligent healthcare. Intelligent healthcare is the application of modern AI technology to make medical services more efficient and accurate [1,2]. This leads to improved diagnosis and treatment for patients and better utilization of medical resources. By alleviating problems such as resource scarcity and uneven distribution [3], intelligent healthcare makes it more feasible for patients to access high-quality medical services and health support [4,5].

With the rise of internet-based medical services and online consultations, people increasingly expect to obtain medical, health, and related guidance via online platforms in a fast and convenient way. Within this context, medical question answering (QA) has gradually become one of the most important tasks in intelligent healthcare. However, hallucinations in LLMs and the uneven quality of online information make it difficult for users to obtain timely, accurate, and reliable medical knowledge. Unlike general-domain question answering, medical question answering requires a high degree of factual accuracy and domain expertise, as it is closely tied to information retrieval, clinical research, and even downstream decision making. Medical topics often involve complex language and rare medical terminology; in many real-world online scenarios, fragmented medical content can also lead to incomplete or ambiguous key phrases in user queries [6]. As a result, incorrect or imprecise answers may cause misdiagnosis, delayed treatment, or other serious consequences. These characteristics pose several new challenges for applying LLMs in medical settings.

To begin with, the medical knowledge system is vast and highly specialized, and existing relatively small-parameter models often struggle to guarantee reliable medical reasoning. The medical domain encompasses extensive and deeply structured knowledge, including disease mechanisms, drug responses, diagnostic procedures, and treatment pathways, all of which depend heavily on precise logic and domain-specific terminology. Even though LLMs possess strong language understanding capabilities, their internal parameters are not specifically optimized for medical scenarios. When facing rigorous medical questions, models may exhibit incomplete understanding, insufficient recall of relevant information, or logically weak explanations. Moreover, the knowledge stored inside an LLM is implicitly encoded in its parameters, making it difficult to provide explicit, verifiable evidence chains. This lack of traceability limits the trustworthiness of LLM outputs in safety-critical medical applications.

Moreover, medical applications impose strict requirements on factual correctness. A key deficiency of LLMs is their tendency to generate fluent but factually incorrect content when evidence is missing or insufficient. In open-domain dialogue, this may only degrade user experience, but in medical scenarios, such hallucinations can directly mislead patients or healthcare providers, potentially leading to dangerous outcomes. Therefore, relying solely on end-to-end LLM generation, without explicit grounding in trustworthy medical knowledge, is inadequate for real-world clinical or health-consultation tasks.

Finally, in medical scenarios, patients’ questions are typically short, colloquial, and highly abstract. When generating answers, LLMs often focus on the surface form of the question and may fail to capture the underlying intent or the truly central medical concern. In addition, LLMs commonly exhibit verbosity bias and overgeneralization: they tend to produce long, exhaustive answers that cover many possible aspects rather than focusing on the core issue implied in the question. For a user seeking clear and targeted medical advice, such overly broad answers increase cognitive burden and make it harder to extract the truly relevant information. Consequently, the generated answers frequently fail to capture the user’s core intent.

To enable the model to reason over structured medical knowledge rather than relying on opaque parameter memory, we construct a domain-specific medical knowledge graph database. It contains approximately 40,000 entities, 260,000 relations, and 400,000 triples. Core medical concepts such as diseases, drugs, examinations, symptoms, and clinical guidelines are represented as structured entities and relations. We also employed an embedding model to encode all entities and relations in the graph into vector representations, enabling subsequent retrieval, reasoning, and prediction. In the remainder of this paper, this knowledge graph is referred to as the Global Knowledge Graph to avoid conceptual confusion. The underlying data are collected from publicly available medical websites and open-access medical information sources. Basic data cleaning is applied to reduce obvious noise and inconsistencies. The knowledge graph is not intended to serve as a comprehensive or authoritative medical knowledge base, but rather as a structured resource to support reasoning in our framework.

Building upon this structured foundation to address these limitations, we propose Co-MedGraphRAG, a medical question-answering framework that combines knowledge graph reasoning with collaborative large–small model inference. The key idea is to let the model start from the medical question itself, construct a question-specific knowledge graph with explicitly marked missing entities, and then use real medical knowledge to complete and refine this graph. The completed graph is then used as external evidence to guide and constrain answer generation.

This work focuses on the following three research questions:

• RQ1: How can large language models be effectively grounded in structured medical knowledge to improve alignment with user intent in medical question answering?
• RQ2: How can implicit or missing variables in medical questions be explicitly identified and resolved to improve structural grounding and interpretability in medical reasoning?
• RQ3: How can large–small model collaboration be leveraged to balance reasoning accuracy and interpretability in medical question answering?

The main contributions, corresponding to the above research questions, are summarized as follows:

• We propose Co-MedGraphRAG, a framework that shifts the paradigm from “passive retrieval” to “active, question-specific knowledge construction.” By integrating knowledge graph reasoning with large–small model collaboration, this approach is designed to mitigate intent misalignment and improve factual grounding in medical LLM applications by grounding generation in structured knowledge.
• We introduce a method for constructing an “incomplete” knowledge graph containing explicit “none” placeholders. By explicitly modeling the implicit variables within a question, this mechanism forces the model to focus on core constraints, thereby effectively overcoming the loss of focus caused by verbosity bias and overgeneralization.
• We design a collaborative workflow strategy that leverages the complementary strengths of large and small models. This strategy combines the efficient probabilistic prediction of small models on the knowledge graph with secondary semantic reweighting by a large language model. This integration achieves a synergy between structured probability and semantic understanding, ensuring reasoning precision while maximizing the advantages of different model architectures.

The remainder of this paper is organized as follows. Section 2 reviews related work on medical question answering, retrieval-augmented generation, knowledge graph reasoning, and large–small model collaboration. Section 3 describes the proposed Co-MedGraphRAG framework and its core components. Section 4 presents the experimental setup, evaluation metrics, and comparative results on multiple medical datasets. Section 5 discusses medical safety, clinical risks, and legal considerations related to the proposed approach. Section 6 outlines the limitations of this work and directions for future research. Finally, Section 7 concludes the paper.

2. Related Work

2.1. Medical Question Answering

Medical Question Answering (MedQA) aims to provide reliable health consultations and clinical decision support in real-world medical scenarios. Early research primarily relied on rule-based matching and information retrieval [7,8]. For instance, Jain et al. [7] proposed a rule-based architecture that constructed question classification taxonomies and answer extraction patterns to match content within literature. Similarly, Lee et al. [8] combined retrieval with text analysis, extracting key phrases to identify sentences sharing common linguistic structures with definitions. While these methods achieved high precision in specific narrow domains, they were limited by high maintenance costs and an inability to handle open-domain questions, linguistic diversity, or cross-modal information.

With the widespread application of deep learning in NLP, MedQA evolved from rule-based matching to neural semantic modeling. Researchers utilized pretrained word embeddings coupled with Convolutional Neural Networks (CNNs) or Bidirectional Long Short-Term Memory networks (BiLSTM) to extract context features, incorporating attention mechanisms to rank candidate answers. Zhang et al. [9] introduced a character-level multi-scale CNN for Chinese medical QA, while others employed Recursive Neural Networks (RNNs) [10,11]. Abacha and Demner-Fushman [12] proposed a method based on Recognizing Question Entailment (RQE), and Yan et al. [13] developed a deep ranking recursive autoencoder. Although these deep models outperformed traditional retrieval methods, they still faced challenges regarding data scarcity and limited interpretability.

The advent of Transformer architectures and pretrained language models, such as BERT and its biomedical variant BioBERT [14], further advanced the field. BioBERT, after fine-tuning, achieved state-of-the-art results on biomedical QA tasks. Subsequently, reinforcement learning agents [15] and paraphrase-based frameworks [16] were introduced to refine question formulations. Following the rise of Large Language Models (LLMs), MedQA witnessed breakthroughs in semantic understanding and generation. However, despite their reasoning capabilities, LLMs face significant challenges in medical settings: they are prone to hallucinations when factual support is absent, and their knowledge is implicitly stored in parameters, making it difficult to provide verifiable reasoning paths. Consequently, relying solely on end-to-end LLM generation often fails to meet the strict reliability and transparency requirements of clinical applications.

2.2. Retrieval-Augmented Generation and Knowledge Graph QA

Retrieval-Augmented Generation (RAG) enhances LLMs by linking them to external resources to improve accuracy. Khandelwal et al. [17] demonstrated that accessing relevant training data significantly boosts performance. Lewis et al. [18] formalized the RAG framework, retrieving relevant texts to ground generation. Subsequent works like REALM [19] and Fusion-in-Decoder [20] integrated retrieval into pre-training or decoding stages. However, text-based RAG relies on unstructured text, which often contains noise and struggles with multi-hop reasoning or ambiguous medical entity relationships, limiting its effectiveness in complex diagnostic scenarios.

To address the need for structured reasoning, Knowledge Graphs (KGs) have become essential. Combining RAG with KGs can enrich retrieval with structured knowledge and complementary evidence sources. Xu et al. [21] proposed MEGA-RAG to mitigate hallucinations in public health via multi-evidence guidance.

Despite these advancements, existing methods generally adopt a passive retrieval strategy, relying heavily on the completeness and connectivity of predefined static graphs. When facing noisy, disconnected graphs or incomplete paths, these methods often fail to retrieve relevant evidence. Furthermore, they typically treat KGs merely as a static retrieval index rather than integrating them deeply with the LLM’s reasoning process. Consequently, when graph coverage is insufficient, the model may still hallucinate.

2.3. Collaborative Large–Small Model Frameworks

Large language models possess remarkable few-shot capabilities, but their deployment in real-world scenarios is hindered by immense parameter sizes (often exceeding hundreds of billions) and high inference costs [22,23]. Relying on cloud-based APIs also introduces latency and privacy concerns. To address these issues, researchers have explored knowledge distillation and collaborative frameworks. Xu et al. [24] proposed WizardLM using Evol-Instruct to fine-tune small models with complex instructions generated by LLMs. Ranaldi and Freitas [25] transferred reasoning capabilities via Chain-of-Thought (CoT) alignment.

Beyond unidirectional transfer, Chen et al. [26] introduced a “Data Shunt” paradigm, dynamically routing simple samples to small models and difficult ones to cloud LLMs based on confidence scores. Building on this, they further proposed a framework [27] incorporating prompt pruning and two-stage confidence distillation to enable bidirectional enhancement. Similarly, SuperICL [28] utilized fine-tuned small models as plug-ins to augment LLM in-context learning.

However, a significant trade-off remains in existing methods: small models excel in efficiency but lack semantic depth, while large language models offer powerful reasoning but suffer from high costs and hallucination risks. How to achieve deep collaboration around medical knowledge graphs to balance efficiency, reliability, and interpretability remains an open challenge.

Co-MedGraphRAG differs by integrating the active construction of a question-specific knowledge graph with retrieval and reasoning over the Global Knowledge Graph. Instead of relying solely on static indices, it adopts large–small model collaboration to jointly ensure structural grounding and semantic precision in medical reasoning.

3. Methodology

3.1. Overview

As shown in Figure 1, the Co-MedGraphRAG comprises three primary stages: (1) construction of an incomplete question-specific knowledge graph, (2) knowledge graph completion via hybrid reasoning, and (3) augmented answer generation. In the following sections, we detail the workflow of each module within this framework.

Figure 1. Overall framework of Co-MedGraphRAG.

3.2. Construction of an “Incomplete” Question-Specific Knowledge Graph

In this initial stage, the medical question serves as the sole input and the LLM constructs a question-specific medical knowledge graph. This graph $G_q^{\left(0\right)}$ is represented as a set of triples, as formulated in Equation (1):

$$G_q^{\left(0\right)}=F_{\mathrm{kg}}^{\mathrm{LM}}\left(q\right)={\left\{(h_i,r_i,t_i)\right\}}_{i=1}^{N_q}$$

(1)

where q denotes the medical question, $F_{\mathrm{kg}}^{\mathrm{LM}}$ represents the KG construction process, and $(h_i,r_i,t_i)$ corresponds to the head entity, relation, and tail entity, respectively. To explicitly represent the “unknown information” inherent in the question, we introduce a special placeholder “none”, denoted by the symbol ⊥. The entity space is defined as shown in Equation (2):

$$h_i,t_i\in V\cup \{\perp \},r_i\in R,$$

(2)

where V is the set of all known medical entities, R is the relation set defined in the Global Knowledge Graph, and ⊥ indicates pending entities.We constrain ⊥ to appear exclusively in the head or tail positions, manifesting in the following two forms (Equation (3)):

$$(h_i,r_i,t_i)\in (V\times R\times \perp )\cup (\perp \times R\times V).$$

(3)

A key premise of our approach is that most meaningful medical questions imply at least one unknown variable. Therefore, we impose a constraint on the generation process: the constructed triple set must contain at least one triple with the placeholder ⊥. The overall workflow of this initial construction phase is visually depicted as Stage 1 in Figure 1.

3.3. Structure-Aware Retrieval Under Relation Constraints and Small-Model Prediction

For every triple containing a placeholder, such as $(h,r,\perp )$ or $(\perp ,r,t)$, it executes a structured query over the Global Knowledge Graph. The query utilizes a “fixed relation + known entity” pattern to retrieve all candidates satisfying the structural constraints.

However, in real-world scenarios, the entity names extracted by the LLM may not perfectly match the canonical names in the database. To address this, we design a two-tier matching strategy.

Tier 1: Alias Matching. Since entities in the knowledge graph are associated with multiple aliases, we first attempt to match the query entity against these predefined aliases.

Tier 2: Vector Similarity Matching. If alias matching fails, we utilize vector embeddings. Specifically, we compute the cosine similarity between the query entity and candidate entities connected by the given relation within the Global Knowledge Graph, selecting the most similar entity as the matched entity:

$$\widehat{e}=arg\underset{v\in V_r}{max}\mathrm{sim}\left(\varphi \left(e_q\right),\varphi \left(v\right)\right).$$

(4)

Equation (4) defines this selection process, where $e_q$ denotes the query entity, $V_r$ represents the subset of candidate entities in the graph that share the relation r, $\varphi (·)$ is the embedding function, and $\mathrm{sim}(·)$ denotes cosine similarity. The matched entity $\widehat{e}$ is then used to re-execute the structured retrieval, yielding a candidate set C.

3.3.1. Small Model Training

As a design choice, we adopt the Long Short-Term Memory (LSTM) network as the reasoning engine to model structural dependencies within knowledge graph triples. The rationale is primarily based on the following factors:

• Sequential Characteristic: Although a triple itself represents a static fact, the reasoning process over a knowledge graph is inherently sequential, as it involves traversing ordered entity–relation paths from one entity to another. Previous work has shown that LSTM networks are effective in encoding sequences of entities and relations to capture path-level dependency patterns [29]. Inspired by this, we adopt an LSTM network to model the conditional dependency between an entity, a relation, and its connected counterpart. By feeding the embeddings of the entity–relation pair into the LSTM, the network produces a structured transition representation that supports effective prediction of the target entity.
• Efficiency and Low Latency: LSTM is characterized by rapid computation and low overhead. It can quickly calculate the probabilities of large-scale candidate entity sets, demonstrating the characteristics of high efficiency and low latency.

After completing vector embedding, we extract all triples from the Global Knowledge Graph and create masked training samples by masking one entity in each triple. The trained LSTM model functions by accepting the embeddings of the known entity and relation as input to predict the vector representation of the missing entity. Subsequently, we employ cosine similarity to calculate the alignment between this predicted vector and the embeddings of the entities in the candidate set.

The specific training configuration of the LSTM is summarized as follows. Training samples are generated by masking one entity in each triple of the Global Knowledge Graph, resulting in twice the number of original triples as training instances (approximately 800,000 samples). Entity and relation embeddings are fixed during training and have a dimensionality of 1024, consistent with the embedding model used in the retrieval stage. The LSTM is configured with two stacked layers and a hidden size of 512. Model training is performed using the Adam optimizer with a learning rate of $1\times {10}^{-3}$ and a batch size of 256, with candidate triples sampled uniformly.

3.3.2. Probability Estimation over Candidate Entities

Upon obtaining the candidate set C, we input the context (the known entity and relation) into the trained LSTM to generate a predicted target vector. The compatibility score $s\left(c\right)$ for each candidate is then derived by computing the cosine similarity between the predicted vector and the vector of each candidate $c\in C$. These scores are normalized via a Softmax function to form a probability distribution (Equation (5)):

$$p\left(c\right)={\displaystyle \frac{exp\left(s\right(c\left)\right)}{{\sum }_{c^{\prime }\in C}exp\left(s\left(c^{\prime }\right)\right)}},c\in C,$$

(5)

In Equation (5), $s\left(c\right)$ denotes the cosine similarity score calculated for candidate c. This step provides a quantitative basis for the subsequent semantic selection by the large language model. It should be noted that the Softmax normalization in Equation (5) is performed over the restricted candidate set C. Accordingly, the resulting values should be interpreted as relative ranking scores within C, rather than globally calibrated probabilities over the entire entity space.

3.4. Semantic Probability Fusion and Graph Completion

While the LSTM provides a structural probability distribution, statistical likelihood does not always equate to semantic optimality. In complex medical contexts, a lower-probability entity might align better with diagnostic logic. Therefore, we further employ the large language model to perform a secondary semantic scoring over the candidates.

For each triple containing a placeholder, we feed the following information into the LLM: the original question q, the incomplete triple $(h,r,\perp )$, the candidate set C, and the probability distribution $p\left(c\right)$. The LLM then assesses the candidates based on semantic relevance and generates a new probability distribution, as defined in Equation (6):

$$P_{\mathrm{LLM}}=F_{\mathrm{LM}}^{\mathrm{comp}}\left(q,(h,r,\perp ),C,P_{\mathrm{LSTM}}\right),$$

(6)

where $P_{\mathrm{LSTM}}$ denotes the input probabilities from the small model, and $P_{\mathrm{LLM}}$ represents the semantic probability distribution generated by the large language model.

To combine the structural guidance of the LSTM with the semantic reasoning of the LLM, we compute the final calibrated probability $P_{\mathrm{final}}$ using a weighted fusion strategy (Equation (7)):

$$P_{\mathrm{final}}\left(c\right)=\alpha ·P_{\mathrm{LLM}}\left(c\right)+(1-\alpha )·P_{\mathrm{LSTM}}\left(c\right),$$

(7)

In Equation (7), $P_{\mathrm{final}}\left(c\right)$ denotes the final calibrated probability score for candidate entity c. $P_{\mathrm{LLM}}\left(c\right)$ and $P_{\mathrm{LSTM}}\left(c\right)$ represent the probability distributions derived from the semantic reasoning of the Large Language Model and the structural prediction of the LSTM, respectively. The hyperparameter $\alpha \in [0,1]$ serves as a weighting coefficient to balance the influence of these two modalities. In all experiments of our framework, $\alpha$ is fixed to 0.4. The final candidate set S is then derived by selecting the top-ranked entities. Consequently, for the finalized set $S=\{t_1,t_2,\dots ,t_n\}$, the single incomplete triple $(h,r,\perp )$ is expanded into n complete triples as illustrated in Equation (8):

$$\{(h,r,t_1),(h,r,t_2),\dots ,(h,r,t_n)\}.$$

(8)

This design ensures the completed knowledge graph is not only structurally sound but also semantically interpretable and extensible. Once all incomplete triples are processed, we obtain a fully completed, question-specific medical knowledge graph. The workflows described in Section 3.3 and this section collectively constitute Stage 2 in Figure 1.

3.5. Answer Generation and Evaluation

Medical answers must be factually accurate, logically coherent, and readable. In the final stage, we input both the original question q and the completed knowledge graph into the LLM to generate the final answer, using the graph as structured external evidence.

To mitigate the bias of self-evaluation, we employ a more powerful model with a larger parameter scale, GPT-4o [30], to assess the generated answers. This strategy is employed to address the high costs and subjectivity associated with manual evaluation, offering a scalable and explainable mechanism to approximate human preferences. This methodology aligns with the “LLM-as-a-Judge” paradigm validated by Zheng et al. [31], which establishes that advanced LLMs can effectively approximate human judgment with high agreement rates, thus ensuring a reliable assessment of model performance.

The evaluation covers five dimensions, each scored between 0 and 1. These metrics were designed by referencing clinical evaluation perspectives, serving as proxy indicators for real-world medical utility: (1) Medical Accuracy, (2) Completeness, (3) Practicality, (4) Relevance, and (5) structured knowledge support. A hierarchical weighting scheme is applied to calculate the comprehensive score, with evaluation criteria and specific weights detailed in Section 4.3. The answer generation process described herein corresponds to Stage 3 in Figure 1, and the workflow of the evaluation stage is illustrated in Figure 2.

Figure 2. Overview of the multi-dimensional evaluation pipeline.

3.6. Prompt Design and Inference Configuration

For question-specific knowledge graph construction (Section 3.2), the LLM is prompted with the original medical question and a predefined set of relation types aligned with the Global Knowledge Graph schema. The generation is explicitly constrained to extract only entities mentioned in the question, without fabricating entities that are not mentioned in the question. When forming triples, the LLM is restricted to generating incomplete triples with either a missing head entity $(\perp ,r,t)$ or a missing tail entity $(h,r,\perp )$; complete triples $(h,r,t)$ are also included when all required entities can be directly identified from the question. At least one incomplete triple containing the placeholder ⊥ is required to ensure explicit modeling of unknown variables. All extracted triples are output in a fixed structured format.

For semantic probability re-weighting (Section 3.4), the LLM is provided with the original medical question, the incomplete triple, the candidate entity set C retrieved from the Global Knowledge Graph, and the structural probability distribution $P_{\mathrm{LSTM}}$ produced by the small model. The LLM is instructed to perform semantic comparison strictly within the given candidate set, without proposing new entities. Its role is to reassess the relative semantic plausibility of candidates under the medical context and to output a probability distribution over C, rather than free-form textual explanations. Specifically, the LLM is instructed to output a probability distribution over the candidate entities that is normalized using a Softmax function, yielding $P_{\mathrm{LLM}}$. During both knowledge graph construction and semantic re-weighting stages, decoding is performed with a temperature of 0.7.

For final answer generation (Section 3.5), the LLM takes the original medical question and the completed question-specific knowledge graph as structured external evidence. The generation is guided by principles of graph-grounded reasoning, discouraging unsupported speculation beyond the provided knowledge graph. To balance factual consistency and linguistic fluency, the decoding temperature is set to 0.3 at this stage.

4. Experiments

4.1. Datasets

To cover more medical question-answering scenarios, we conduct experiments on three publicly available datasets: (1) Chinese Medical Dialogue, (2) CMQA, and (3) Huatuo. The Chinese Medical Dialogue dataset [32] comprises patient inquiries across diverse medical departments. It incorporates colloquialisms and informal descriptions that are common in real-world scenarios. The CMQA dataset [33] is a publicly available Chinese medical question-answering corpus covering a wide range of common diseases and drug-related inquiries. The Huatuo dataset [34], derived from Li et al.’s “Huatuo-26M,” represents one of the largest Chinese medical corpora currently available. It spans multiple medical domains—including diseases, symptoms, treatments and drug information. The original corpus contains over 26 million pairs and we utilize the encyclopedia subset (approximately 360 k). Additional details of the datasets used in our experiments are summarized in Table 1 and all datasets were subjected to basic preprocessing and evaluation of data by removing inconsistent data.

Table 1. Summary of the medical datasets, including total size, remaining size (after preprocessing), collection period, and medical content types.

Dataset	Total Size	Remaining Size	Collection Period	Content Types
Chinese Medical Dialogue	790,000	610,000	2019–2020	Inquiries involving multiple clinical specialties
CMQA	268,000	223,000	2020	General Chinese medical QA
Huatuo	364,000	352,000	2022–2023	Medical QA from Wikipedia and medical articles

4.2. Baselines

To verify the effectiveness of the proposed approach, we compare Co-MedGraphRAG with several mainstream Retrieval-Augmented Generation (RAG) frameworks, including NaiveRAG [35], GraphRAG [36], and LightRAG [37].

For a fair comparison, we conduct experiments using two Large Language Models (LLMs), Qwen2.5-7B and GLM4-9B [38], and employ Bge m3-embedding [39] as the embedding model.

4.3. Evaluation Metrics

Following the multi-dimensional evaluation paradigm of Hu et al. [40], we define five complementary evaluation dimensions as follows:

• Medical Accuracy: Evaluates whether the answer conforms to clinical knowledge, avoids factual errors, and uses medical terminology correctly.
• Completeness: Measures whether the answer sufficiently expands and elaborates on the question (e.g., etiology, symptoms, treatment, prevention).
• Relevance: Assesses whether the generated content directly addresses the user’s specific query and remains focused on the core problem.
• Practicality: Evaluates the utility of the answer, specifically whether it offers actionable guidance and considers real-world feasibility.
• Structured Knowledge Support: Measures the extent to which the answer is grounded in structured or semi-structured knowledge, rather than being purely generated from the model’s internal parameter memory. This metric is designed to accommodate different forms of structured or semi-structured knowledge representations, and evaluates whether such structured evidence is effectively reflected in the generated answer.

Each dimension is scored on a scale from 0 to 1, employing a specific granular scoring rubric: 0.2 indicates severe deficiencies, irrelevance, or dangerous hallucinations; 0.4 indicates significant errors or major omissions; 0.6 indicates partial compliance, where the core intent is met but noticeable inaccuracies remain; 0.8 indicates substantial quality with only minor issues or imperfections; and 1.0 indicates a flawless answer.

To calculate the overall score, we assign specific weights to each dimension based on their relative importance for medical question answering. The weights are assigned as follows: Medical Accuracy ($0.3$), Completeness ($0.25$), Relevance ($0.2$), Practicality ($0.15$), and Structured Knowledge Support ($0.1$).

Furthermore, we assessed the agreement between the LLM judgments and human evaluations to empirically validate the reliability of the LLM evaluator. We randomly sampled 30 generated responses from each of the three datasets and invited two independent annotators with graduate-level biomedical backgrounds to conduct a manual evaluation using the identical scoring rubric. We calculated the Spearman rank correlation coefficient ($\rho$) between the GPT-4o scores and the human consensus scores. The results showed positive correlations across all five evaluation dimensions ($\rho =0.47,0.41,0.51,0.46,$ and $0.39$ for Medical Accuracy, Completeness, Relevance, Practicality, and Structured Knowledge Support, respectively), and the overall score achieved $\rho =0.44$ ($p<0.001$). This indicates the LLM evaluator is generally consistent with human judgments and can serve as a reasonable proxy for assessment in this study.

4.4. Main Results

The main experimental results are visualized in Figure 3 and Figure 4, and detailed in Table 2 and Table 3.

Figure 3. Five evaluation metrics across three datasets, including the Overall score. We refer to the five metrics as Med Acc, Comp, Rel, Prac and Struct Sup. The performance differences between Co-MedGraphRAG and the baselines are statistically significant ($p<0.05$, Wilcoxon signed-rank test).

Figure 4. The Overall score across three datasets. The comparison is made between baselines and Co-MedGraphRAG. The performance differences between Co-MedGraphRAG and the baselines are statistically significant ($p<0.05$, Wilcoxon signed-rank test).

Table 2. Comparative results on Qwen2.5-7B. Bold text indicates the higher score between the compared methods. The symbol * indicates that the improvement over the baseline is statistically significant ($p<0.05$, Wilcoxon signed-rank test).

	Medical Dialogue		CMQA		Huatuo
	NaiveRAG	Co-Med	NaiveRAG	Co-Med	NaiveRAG	Co-Med
Med. Acc.	0.770	0.853	0.787	0.843	0.813	0.852
Comp.	0.728	0.807	0.788	0.759	0.755	0.793
Rele.	0.942	0.967	0.962	0.952	0.957	0.930
Prac.	0.757	0.879	0.797	0.883	0.773	0.903
Struct.	0.177	0.840	0.213	0.789	0.223	0.773
Overall	0.733	0.867 *	0.766	0.844 *	0.762	0.853 *
	GraphRAG	Co-Med	GraphRAG	Co-Med	GraphRAG	Co-Med
Med. Acc.	0.803	0.853	0.800	0.843	0.817	0.852
Comp.	0.797	0.807	0.805	0.759	0.793	0.793
Rele.	0.895	0.967	0.917	0.952	0.967	0.930
Prac.	0.760	0.879	0.763	0.883	0.765	0.903
Struct.	0.615	0.840	0.640	0.789	0.602	0.773
Overall	0.795	0.867 *	0.803	0.844 *	0.812	0.853 *
	LightRAG	Co-Med	LightRAG	Co-Med	LightRAG	Co-Med
Med. Acc.	0.835	0.853	0.863	0.843	0.807	0.852
Comp.	0.935	0.807	0.923	0.759	0.900	0.793
Rele.	0.855	0.967	0.833	0.952	0.827	0.930
Prac.	0.895	0.879	0.875	0.883	0.858	0.903
Struct.	0.665	0.840	0.712	0.789	0.692	0.773
Overall	0.856	0.867 *	0.859	0.844	0.830	0.853 *

Table 3. Comparative results on GLM4-9B. Bold text indicates the higher score between the compared methods. The symbol * indicates that the improvement over the baseline is statistically significant ($p<0.05$, Wilcoxon signed-rank test).

	Medical Dialogue		CMQA		Huatuo
	NaiveRAG	Co-Med	NaiveRAG	Co-Med	NaiveRAG	Co-Med
Med. Acc.	0.763	0.876	0.807	0.878	0.800	0.900
Comp.	0.663	0.840	0.753	0.816	0.724	0.821
Rele.	0.923	0.932	0.947	0.928	0.953	0.917
Prac.	0.708	0.891	0.785	0.915	0.731	0.867
Struct.	0.193	0.801	0.202	0.765	0.214	0.799
Overall	0.705	0.873 *	0.758	0.867 *	0.742	0.868 *
	GraphRAG	Co-Med	GraphRAG	Co-Med	GraphRAG	Co-Med
Med. Acc.	0.833	0.876	0.803	0.878	0.823	0.900
Comp.	0.800	0.840	0.797	0.816	0.795	0.821
Rele.	0.882	0.932	0.905	0.928	0.865	0.917
Prac.	0.788	0.891	0.785	0.915	0.750	0.867
Struct.	0.623	0.801	0.592	0.765	0.500	0.799
Overall	0.807	0.873 *	0.798	0.867 *	0.781	0.868 *
	LightRAG	Co-Med	LightRAG	Co-Med	LightRAG	Co-Med
Med. Acc.	0.883	0.876	0.855	0.878	0.868	0.900
Comp.	0.915	0.840	0.928	0.816	0.923	0.821
Rele.	0.817	0.932	0.903	0.928	0.863	0.917
Prac.	0.868	0.891	0.835	0.915	0.833	0.867
Struct.	0.746	0.801	0.683	0.765	0.700	0.799
Overall	0.862	0.873 *	0.863	0.867 *	0.859	0.868 *

As illustrated in the radar chart (Figure 3), Co-MedGraphRAG achieves consistent improvements in relevance, practicality, and structured knowledge support compared with mainstream RAG baselines. This indicates that our active graph construction paradigm captures the user’s core intent, ensuring that the generated answers are not only theoretically correct but also directly address the specific medical query. However, a noticeable decline is observed in the Completeness metric. This decline reflects a trade-off introduced by the question-specific knowledge graph design. The model is guided to concentrate on understanding the question’s core intent, so the generated answers tend to focus on it rather than expanding the discussion from multiple medical perspectives. Consequently, this focus imposes certain limitations on the breadth and coverage of the generated information and it results in a disadvantage in the Completeness score.

From the overall scores shown in Figure 4, Co-MedGraphRAG outperforms other baselines on both the Medical Dialogue and Huatuo datasets. On the CMQA dataset, it also maintains competitive performance comparable to the leading baseline, LightRAG. To empirically validate the performance differences, we conducted the non-parametric Wilcoxon signed-rank test on the Overall scores. Taking the best-performing baseline (LightRAG) as a representative instance, the results for the Qwen2.5-7B show significance on the Medical Dialogue ($Z=-4.05,p<0.001$) and Huatuo ($Z=-4.41,p<0.001$) datasets. For the GLM4-9B, statistically significant results were observed across the Medical Dialogue ($Z=-3.90,p<0.001$), CMQA ($Z=-3.45,p<0.001$), and Huatuo ($Z=-3.60$, $p<0.001$) datasets.

4.5. Ablation Studies

To evaluate the contribution of each core component in Co-MedGraphRAG, this section conducts ablation experiments. Table 4 presents the Overall score averaged over Qwen2.5-7B and GLM4-9B.

Table 4. Ablation study on different datasets by scoring, where w/o indicates without the part of the method.

Methods	Medical Dialogue (Overall)	CMQA (Overall)	Huatuo (Overall)
Co-MedGraphRAG	0.870	0.856	0.861
w/o KG Construction & Collaborative Reasoning	0.740	0.733	0.733
w/o LLM Weight Fusion	0.823	0.804	0.797

(1) Removal of KG construction and Collaborative Reasoning: In this setting, we allow LLMs to answer the medical question directly. As shown in the table, performance drops significantly. This indicates that relying solely on the parametric knowledge of LLMs is unreliable for complex medical answering.

(2) Removal of weight fusion: In this setting, we remove the LLM’s semantic contribution and disable the weight fusion mechanism, relying solely on the structural probability distribution calculated from the small model’s predictions. The results indicate a decline in performance compared to the full framework. This highlights the importance of fusing structural probability with the LLM’s semantic reasoning.

5. Discussion

5.1. Ethical Considerations, Safety, and Legal Boundaries

While Co-MedGraphRAG presents a viable framework for enhancing the quality and structured grounding of LLM-generated medical responses, it is explicitly designed as an auxiliary information support approach rather than a substitute for professional medical diagnosis or clinical decision-making. Despite the integration of structured medical knowledge graphs to improve factual grounding, large language models remain probabilistic in nature and may still produce medically plausible yet incorrect information, particularly in complex or ambiguous clinical contexts.

From a medical safety perspective, the proposed approach is not intended for direct or autonomous clinical use. Any real-world application should incorporate a strict human-in-the-loop paradigm, in which all generated content is reviewed, validated, and contextualized by licensed medical professionals prior to clinical decision-making. This requirement is consistent with established practices in clinical decision support and reflects the current limitations of data-driven language models in high-stakes medical settings.

With respect to ethical and legal considerations, clinical responsibility and final decision authority remain with the human practitioner, in accordance with prevailing regulatory and professional standards for AI-assisted medical tools. The proposed approach should therefore be understood as a retrieval- and reasoning-oriented assistant that supports information access and structured interpretation.

5.2. Practical Implications

The practical value of Co-MedGraphRAG is reflected in the following aspects:

Assisting Decision-Making: After the large language model generates a response to a medical query, the output can serve as a reference for healthcare professionals.

Capturing the Key Focus of Medical Queries: The LLM relies on the question-specific knowledge graph as a structured reference. This facilitates the comprehension of the query’s intent.

6. Limitations and Future Work

Despite the promising performance of Co-MedGraphRAG on multiple datasets, showing clear advantages in medical accuracy, relevance, and structured knowledge support, several limitations remain.

• Dependency on Static Knowledge Graphs: The framework relies heavily on the Global Knowledge Graph. The reasoning quality is intrinsically limited by the graph’s coverage and quality. Consequently, knowledge blind spots may exist, particularly concerning rare diseases or rapidly evolving frontier treatments. As a static database, it lacks the ability to self-update with the latest medical findings.
• Limited Reasoning Capacity of the Small Model: While the LSTM performs stably in structural entity prediction, its expressive capacity is constrained compared to Transformer-based models. In scenarios involving complex multi-hop relation chains or deep implicit semantic dependencies, it may struggle to fully model the necessary context.
• Lack of Real-World Validation: Currently, Co-MedGraphRAG operates as a research prototype evaluated primarily on offline datasets. Real-world clinical consultation systems involve complex factors such as user emotions, multi-turn context history, and personalized patient conditions, which are not yet fully reflected in our experimental setting.

Our future work will focus on addressing these issues by: (1) expanding the scale of the medical knowledge graph and implementing mechanisms for regular updates from trusted sources; (2) replacing the LSTM with more powerful lightweight models to enhance complex reasoning; and (3) extending the framework to support multi-turn dialogue and personalized medical QA scenarios.

7. Conclusions

In this paper, we propose Co-MedGraphRAG, a collaborative reasoning framework designed to address three core challenges of Large Language Models in the medical domain: (1) Due to the absence of structured knowledge, there is potential for answer deviations and a lack of traceability in the reasoning process. (2) Insufficient medical knowledge support may lead to factual or terminology errors in medical LLM-based QA. (3) The generated content fails to align with the user’s true intent due to verbosity bias and overgeneralization.

We formulate the QA reasoning pipeline as a three-stage collaborative engineering framework:

1.

Construction: A large language model generates a small-scale, question-specific knowledge graph containing explicit “none” placeholders to define known and unknown variables.

2.

Completion: The structural probability distribution is calculated through knowledge graph reasoning and small model prediction. Then it is fused with the probability distribution generated by the LLM to complete the question-specific knowledge graph.

3.

Generation: The completed graph serves as structured evidence to guide the final answer generation.

Experiments on three medical datasets demonstrate that Co-MedGraphRAG achieves consistent improvements in relevance, practicality, and structured knowledge support compared with mainstream Retrieval-Augmented Generation (RAG) frameworks, while achieving competitive performance in Medical Accuracy. These results indicate that our framework improves alignment with question intent and provides relevant structured knowledge. Overall, this paper presents a collaborative framework that combines the active construction of question-specific knowledge graphs, collaboration between LLMs and small models, and knowledge graph completion to augment LLM generation. This work provides a reference for researchers and developers seeking to design medical question-answering frameworks that integrate structured knowledge and large language models.