Benchmark Evaluation of a Tool-Augmented Large Language Model Agent Using Traditional Asian Medicine Metadata

Abstract

This study evaluated a tool-augmented large language model (LLM) agent system utilizing a traditional Asian medicine (TAM) metadata. A structured information comprising 4780 entries across five entity types (herbs, syndromes, TCM symptoms, modern medicine symptoms, and acupoints) was constructed. Four LLMs (GPT-5.2, GPT-5-mini, Claude Sonnet 4.6, and Claude Haiku 4.5) were evaluated under baseline, non-agentic retrieval (RAG), and tool-augmented (agent) conditions using two public benchmarks: TCMBench (1300 items) and TCMEval-SDT (600 items). McNemar’s test and bootstrap confidence intervals were applied to examine performance differences. Tool augmentation showed a nominally significant improvement for GPT-5-mini on TCMBench term-related items (+4.5 percentage points on paired items, McNemar p = 0.034, 95% CI [+0.9, +8.5] pp), while Claude Sonnet 4.6 showed a nominally significant decline on SDT pathogenesis (−0.038, bootstrap p = 0.006). The agent condition outperformed the non-agentic RAG baseline for GPT-5-mini on TCMBench (agent 85.6% vs. RAG 77.7% vs. baseline 75.4%), suggesting that selective, autonomous tool invocation is more effective than fixed retrieval. Tool usage rates varied substantially across models (2–87%), with moderate usage (30–40%) associated with the most consistent gains. These findings provide empirical evidence on the potential and limitations of metadata-based tool augmentation for LLMs in the TAM domain.

1. Introduction

The rapid advancement of large language models (LLMs) has prompted active exploration of their applicability in the medical domain. LLMs have achieved expert-level performance on medical licensing examinations—for instance, GPT-4 Omni attained a 90.4% accuracy rate on the United States Medical Licensing Examination (USMLE) [1,2], and Med-PaLM 2 reached performance levels comparable to those of medical professionals on the MedQA benchmark [3]. A systematic review and meta-analysis by Moura et al. [4] further confirmed that GPT-4 consistently surpassed passing thresholds across national licensing examinations in medicine, pharmacy, dentistry, and nursing. Beyond examination performance, LLMs have demonstrated potential across diverse medical tasks, including clinical knowledge encoding [5], medical question answering, and clinical decision support. A comprehensive systematic review by Karabacak et al. [6], encompassing 761 studies, documented the exponential growth of LLM evaluation research in clinical medicine—from 1 study in 2019 to 557 in 2024—reflecting the rapidly expanding interest in this field.

In the field of traditional Asian medicine (TAM) and traditional Chinese medicine (TCM), attempts to utilize LLMs for syndrome differentiation, herbal knowledge question answering, and prescription recommendation have been increasing. Wang et al. [7] conducted a scoping review of LLM applications in TCM, identifying promising directions in clinical reasoning, diagnosis, and education. More recently, Wang et al. [8] evaluated the role of LLMs in TCM diagnosis and treatment recommendations using models such as GPT-4o and Qwen, and Zheng et al. [9] investigated approaches to evaluate and improve the syndrome differentiation thinking ability of LLMs. Despite these emerging efforts, systematic benchmark-based evaluation studies in this domain remain in their early stages.

However, LLMs rely on parametric knowledge derived from their training data, which can lead to hallucination—the generation of plausible but factually incorrect content—in tasks requiring precise factual information [10]. The medical domain is particularly vulnerable to such hallucinations; Umapathi et al. [11] developed Med-HALT, a hallucination test specifically designed for medical LLMs, demonstrating that even state-of-the-art models produce medically inaccurate outputs on reasoning and memory-based tasks. In TAM, accurate factual information such as the properties and meridian tropism of herbs, WHO standard codes for acupoints, and definitions of syndromes is clinically important. The limitations of LLMs in this regard have been confirmed through TCMBench, a benchmark for evaluating TCM knowledge [12], which reported that multiple LLMs showed insufficient performance on TCM licensing examination items, with particular room for improvement in tasks requiring accurate reproduction of specialized terminology. Meanwhile, the TCMEval-SDT benchmark provides a systematic framework for evaluating LLMs’ pathogenesis reasoning and syndrome differentiation abilities in clinical reasoning tasks [9]. Chen et al. [13] further demonstrated through a large-scale benchmarking study that LLM performance varies substantially depending on task type, with factual retrieval tasks and complex reasoning tasks showing markedly different patterns.

In this study, we developed a tool-augmented LLM agent system equipped with a TAM metadata database and function-calling-based tools and evaluated the tool-augmentation effect across four LLMs using two public benchmarks. We additionally compared the agent system against a non-agentic retrieval (RAG) baseline, applied statistical significance testing, and conducted confounding and sensitivity analyses. The main contributions of this study are as follows:

• We constructed a structured TAM metadata database (4780 entries across five entity types) with three function-calling tools and evaluated the tool-augmentation effect across four LLMs on 1900 public benchmark items.
• We demonstrated through statistical testing that tool augmentation can improve lightweight models on factual retrieval tasks.
• We conducted a three-way comparison showing that autonomous tool invocation (agent) outperforms fixed retrieval (RAG) on factual tasks.
• We identified substantial model-level variation in tool usage patterns (2–87%) and demonstrated through conditional analysis that models selectively invoke tools on items they find uncertain.

2. Related Work

2.1. RAG and Tool-Augmented LLMs

Approaches to mitigate LLM hallucination in knowledge-intensive tasks include retrieval-augmented generation (RAG) [14] and tool augmentation. Sahoo et al. [15] conducted a systematic review of RAG applications for LLMs in healthcare, highlighting both the potential and remaining challenges. Tool augmentation represents an alternative paradigm, providing LLMs with external tools for database queries, calculations, and API calls [16]. The ReAct framework proposed by Yao et al. [17] formalized this approach by synergizing reasoning and acting, allowing agents to interleave chain-of-thought reasoning with external tool invocations. In the medical domain, structured knowledge integration approaches such as leveraging medical knowledge graphs with LLMs for diagnosis prediction [18] have shown promise. Hoch and Simbeck evaluated retrieval-augmented generation variants for clinical decision support, demonstrating the importance of controlling retrieval behavior to balance accuracy and efficiency [19].

2.2. TAM/TCM Benchmarks and Knowledge Representation

TCMBench [12] provides a comprehensive benchmark for evaluating TCM knowledge across 16 subjects, while TCMEval-SDT [9] offers a framework for assessing syndrome differentiation and pathogenesis reasoning. Li et al. developed a TCM case-based question-answering system integrating LLMs with knowledge graphs [20]. Recent advances in TCM prescription recommendation include reinforcement learning-based approaches [21] and syndrome differentiation-based methods [22]. Guo et al. reviewed the advancing modernization of TCM through artificial intelligence and multimodal data integration [23]. However, no prior study has systematically evaluated tool-augmented LLM agent systems utilizing TAM-specific metadata with statistical rigor.

3. Materials and Methods

3.1. System Architecture

3.1.1. Metadata Database

We constructed a metadata database that structures TAM terminology and metadata for use through LLM function calling. The database comprises five entity types with a total of 4780 entries; the sources and key attributes of each entity are presented in

Table 1. Composition of the metadata database.

Entity Type	Count	Source	Key Attributes
Herbs	698	HERB DB, SymMap	Multilingual names, properties, meridians, classification
Syndromes	233	SymMap	Multilingual names, definition
TCM Symptoms	2285	SymMap	Multilingual names, definition, body part, nature
MM Symptoms	1148	SymMap	Multilingual name, definition, UMLS/MeSH/ICD-10 codes
Acupoints	416	WHO Standard	Multilingual names, WHO code, meridian, location, needling method
Total	4780

.

The herb data (698 entries) were collected from the HERB DB [24] and SymMap [25] databases. HERB is a high-throughput experiment- and reference-guided database of TCM that integrates information on herbs, ingredients, targets, and diseases; it has recently been updated to version 2.0 with expanded clinical and experimental evidence. The herb entries include multilingual names along with attributes such as properties, meridian tropism, drug classification, and use parts. Korean names for all entity types were curated and verified by two Korean medicine experts. Syndrome data (233 entries), TCM symptom data (2285 entries), and modern medicine (MM) symptom data (1148 entries) were collected from the SymMap database, an integrative TCM database that maps relationships between traditional Chinese medicine symptoms, modern medicine symptoms, herbs, and molecular targets. The MM symptom data include UMLS, MeSH, and ICD-10 codes, enabling linkage with modern medical terminology systems.

Acupoint data (416 entries) were constructed based on the World Health Organization (WHO) standard acupoint code system [26]. The WHO standardization project established unified nomenclature and locations for 361 acupoints across 14 meridians [27], providing an internationally recognized framework for acupoint identification. The acupoint entries include attributes such as affiliated meridian, location, point selection method, and needling technique. During preprocessing, entries with suppression flags set in the original data were removed, attribute names were standardized, and data were stored in CSV (comma-separated values) format. At runtime, the data were loaded into pandas DataFrames in memory for search operations.

3.1.2. Function-Calling Tools

Three function-calling tools were designed to enable LLMs to utilize the metadata database. Function calling (also referred to as tool use) is a mechanism provided by LLM APIs that allows models to generate structured requests for external function execution during response generation. Each tool is automatically invoked upon the LLM’s request and returns search results in JSON (JavaScript Object Notation) format. The tool definitions were provided to the LLM through the API’s function-calling interface, allowing the model to autonomously decide when and which tool to invoke based on the input query.

(1)

search_entity: Searches the TAM metadata database for terms. Herbs, acupoints, syndromes, and symptoms can be searched by Korean, Chinese, English, Pinyin, or WHO code, with optional category specification to limit the search scope. The function accepts three parameters: term (required, the search query string), category (optional, one of “herb”, “syndrome”, “tcm_symptom”, “mm_symptom”, or “acupoint”), and limit (optional, maximum number of results to return, default 5).

(2)

get_entity_info: Returns the complete metadata of a specific entity, including detailed information such as herb properties and meridian tropism, acupoint location and needling technique, and syndrome definitions. The function accepts two required parameters: term (the entity name or identifier) and category (the entity type).

(3)

get_multilingual_names: Returns the multilingual names (Korean, Chinese, English, Pinyin, and Latin) of a specific entity. The function accepts two required parameters: term and category.

3.1.3. LLM Agent Pipeline

When a user query (or benchmark item) is input, the LLM analyzes the query and, under the tool-augmented condition, optionally queries metadata through function calling before generating the final response. The overall pipeline operates as follows: (1) the benchmark item is formatted as a user message along with a system prompt; (2) the LLM processes the input and may generate one or more function-calling requests; (3) each requested function is executed against the metadata database, and the results are appended to the conversation context; (4) the LLM generates the final response incorporating both its parametric knowledge and any retrieved metadata.

The system prompt for the tool-augmented condition included: (1) role designation as a TAM expert, (2) information about the types and number of available metadata entries (e.g., “You have access to a TAM metadata database containing 698 herbs, 233 syndromes, 2285 TCM symptoms, 1148 MM symptoms, and 416 acupoints”), and (3) tool usage principles—“Tools are supplementary; first derive an answer from your own knowledge and only verify uncertain facts with tools; no more than three tool calls per item are recommended.” Under the baseline condition, only the TAM expert role was designated without any tool-related content. For each item, the LLM was allowed up to three iterations for tool calls, and the maximum number of generated tokens was set to 2048. Temperature was set to 0 for all models.

3.2. Benchmarks

Two public benchmarks in the TAM domain were used to evaluate the system.

3.2.1. TCMBench

TCMBench [12] is a public benchmark for evaluating TCM knowledge. In this study, we used 1300 expert-verified items. The items are five-choice multiple-choice questions written in Chinese, spanning 16 subjects including Basic Theory of TCM, TCM Diagnostics, Chinese Materia Medica, Formulas, and Acupuncture.

We classified items from five subjects directly related to the metadata database content (Basic Theory of TCM, TCM Diagnostics, Chinese Materia Medica, Formulas, and Acupuncture) as ‘term-related’ items (331 items) and items from the remaining 11 subjects as ‘other’ items (969 items). This classification was designed to analyze whether the tool-augmentation effect differs by task type—specifically, whether items requiring factual recall of metadata-relevant terminology show greater benefit from tool augmentation compared to items requiring broader clinical knowledge.

3.2.2. TCMEval-SDT

TCMEval-SDT [9] is a benchmark for evaluating syndrome differentiation ability on clinical cases. Each of 300 clinical cases requires performance of two tasks—pathogenesis reasoning and syndrome identification—yielding a total of 600 items. Each item has multiple correct answers in an open-ended format. Unlike TCMBench’s factual recall format, TCMEval-SDT requires comprehensive clinical reasoning, integrating patient symptoms, medical history, and theoretical frameworks to arrive at correct diagnoses. This contrast in task requirements provides an opportunity to evaluate the differential effects of tool augmentation across distinct cognitive demands.

3.3. Experimental Design

3.3.1. Model Configuration

Four LLMs from two API providers were selected (Supplementary Table S1): one large and one lightweight model from each provider, forming large–lightweight model pairs to enable comparison of tool-augmentation effects by model scale. OpenAI’s GPT-5.2 (large, $1.75/$14.00 per million input/output tokens) and GPT-5-mini (lightweight, $0.25/$2.00), and Anthropic’s Claude Sonnet 4.6 (large, $3.00/$15.00) and Claude Haiku 4.5 (lightweight, $0.80/$4.00) were selected.

3.3.2. Experimental Conditions

Three conditions were compared:

(1)

Baseline: The LLM generates responses using only its parametric knowledge without tools. Function calling is deactivated; no tool definitions are included in the API request.

(2)

Tool-augmented (Agent): Three metadata retrieval tools are provided to the LLM through the function-calling API interface, which autonomously determines whether to invoke tool calls. Up to three tool-call iterations per item are allowed.

(3)

Non-agentic retrieval (RAG): For each benchmark item, the search_entity tool is automatically invoked with the question text as the query, and the top-5 retrieved metadata entries are prepended to the prompt as context. The LLM receives this retrieved context but cannot make additional tool calls. This condition serves as a baseline to isolate the effect of autonomous tool-calling from the effect of the metadata itself. The RAG condition was evaluated for GPT-5-mini, the model that showed the largest agent-condition improvement.

The total experimental scale for the main experiment comprised 1900 items × 4 models × 2 conditions = 15,200 API calls, plus 1900 items × 1 model for the RAG condition, with 0.5-s intervals between calls to comply with rate limits.

3.3.3. Prompt Optimization

A pilot experiment (80 items, 20 per benchmark section) revealed that unrestricted tool access led to excessive invocations in Claude Sonnet 4.6, degrading SDT performance. Accordingly, the system prompt for the tool-augmented condition included usage guidelines instructing models to derive answers independently first and use tools only to verify uncertain facts, with a recommended limit of three calls per item. The 80 pilot items were retained in the main experiment; a sensitivity analysis confirmed their inclusion did not materially affect results (maximum delta difference: 0.85 pp; Supplementary Table S2).

3.4. Evaluation Methods

3.4.1. TCMBench Scoring

TCMBench was scored using single-answer accuracy. To extract the correct answer alphabet (A–E) from the LLM’s free-text response, multi-stage regular expression patterns were applied in sequence: (i) explicit answer markers (“Answer:”, “答案:” etc.), (ii) emphasis markers (bold or underlined single letters), (iii) single alphabets on the first line, and (iv) end-of-response search, thereby maximizing the extraction success rate. This multi-stage approach achieved a 99.7% extraction rate (1296 out of 1300 items). Accuracy was calculated as the proportion of items where the predicted answer matched the correct answer.

3.4.2. TCMEval-SDT Scoring

TCMEval-SDT was evaluated using the multi-answer F1 score. For each item, the LLM’s response was parsed to extract predicted answer terms, which were then compared against the reference answer set. Precision (|predicted ∩ correct|/|predicted|) and recall (|predicted ∩ correct|/|correct|) were calculated between the predicted and correct answer sets, and the F1 score was computed as their harmonic mean. This metric accounts for both the completeness (recall) and precision of the model’s predictions, providing a balanced evaluation for items with multiple correct answers. We acknowledge that this string-matching approach may not fully capture synonyms or semantically equivalent expressions; however, the consistent application of the same scoring across all conditions ensures valid within-study comparisons.

3.4.3. Tool Usage Statistics

To analyze tool usage patterns under the tool-augmented condition, the tool usage rate (proportion of items with at least one tool call) and average number of tool calls per item were calculated for each model. Additionally, we recorded which specific tools were invoked (search_entity, get_entity_info, or get_multilingual_names) to characterize the tool selection strategies of different models.

3.5. Statistical Analysis

To assess the statistical significance of performance differences between baseline and tool-augmented conditions, two complementary approaches were employed. For TCMBench (binary correct/incorrect outcomes on paired items), McNemar’s test was applied. For each model–section pair, we constructed a 2 × 2 contingency table of item-level outcome changes: b (baseline wrong → tool correct) and c (baseline correct → tool wrong). Only items with valid answer extractions under both conditions were included in the paired analysis. For all task sections (including SDT continuous F1 scores), nonparametric bootstrap confidence intervals were computed. For each model–section pair, 10,000 bootstrap resamples of item-level score differences (tool − baseline) were drawn with replacement, and 95% percentile confidence intervals were derived. A two-sided bootstrap p-value was calculated as twice the proportion of resamples on the opposite side of zero from the observed mean delta. All significance tests were conducted at α = 0.05. As this is an exploratory study with 16 model–section comparisons, the familywise error rate (FWE) under the global null hypothesis would be approximately 0.56. Results that reached p < 0.05 but did not meet the Bonferroni-corrected threshold (α = 0.003) are referred to as nominally significant throughout this paper.

3.6. Cost Calculation

API costs were calculated using the formula: cost = (input_tokens/1,000,000 × input_price) + (output_tokens/1,000,000 × output_price), where input_price and output_price correspond to the per-million-token rates listed in Supplementary Table S1. Under the tool-augmented condition, costs include overhead from tool definition tokens (included in every API call) and tool result tokens (appended to the conversation context after each tool invocation).

3.7. Experimental Setup

Experiments were conducted in a Python 3.11 (Anaconda) environment. Key packages included the OpenAI Python SDK (version 1.82.0), Anthropic Python SDK (version 0.84.0), pandas 2.2.3, and rapidfuzz 3.14.3. The metadata database was loaded from CSV files into pandas DataFrames; no separate vector database or embedding model was used. All API calls were made sequentially with 0.5-s inter-call delays.

4. Results

4.1. Overall Performance Comparison

4.1.1. TCMBench Performance

The subject-level performance of the four models on TCMBench is shown in Figure 1. Under the baseline condition, Claude Sonnet 4.6 achieved the highest accuracy with 89.4% on term-related items and 87.4% on other items, followed by GPT-5.2 at 84.3% (term) and 78.8% (other). The lightweight models, GPT-5-mini (75.4% term, 71.0% other) and Claude Haiku 4.5 (73.7% term, 70.3% other), showed 10–17 percentage points lower accuracy compared to the large models.

Under the tool-augmented condition, GPT-5-mini showed the most pronounced performance change, improving from 75.4% to 85.6% (+10.2 percentage points) on term-related items and from 71.0% to 76.7% (+5.7 percentage points) on other items. Claude Sonnet 4.6 improved from 89.4% to 91.7% (+2.3 percentage points) on term-related items. In contrast, GPT-5.2 showed slight decreases under tool augmentation (−3.2 percentage points on term, −2.2 on other), while Claude Haiku 4.5 showed marginal changes (+1.1 on term, +0.4 on other).

4.1.2. TCMEval-SDT Performance

Under the baseline condition, Claude Sonnet 4.6 showed the highest performance with 0.652 on pathogenesis and 0.671 on syndrome, followed closely by GPT-5.2 at 0.611 (pathogenesis) and 0.663 (syndrome). Under the tool-augmented condition, SDT performance showed slight decreases or no change for most models. Only GPT-5-mini improved on the pathogenesis task (from 0.474 to 0.508, +0.034), while declining on the syndrome task (from 0.517 to 0.478, −0.039). Claude Sonnet 4.6 showed declines of −0.038 (pathogenesis) and −0.021 (syndrome) (Figure 2).

4.2. Tool-Augmentation Effect

We observed tool-augmentation effects that varied substantially by task type and model (Figure 3; Supplementary Table S3). On TCMBench term-related items, GPT-5-minishowed a nominally significant improvement (+4.5 pp on paired items; McNemar p = 0.034, bootstrap 95% CI [+0.9, +8.5] pp), while no other model reached nominal significance. The +4.5 pp delta from paired analysis differs from the +10.2 pp overall accuracy gain (Section 4.1) because the paired analysis excludes items where answer extraction failed in either condition; GPT-5-mini’s high extraction failure rate (26.1%) under tool augmentation substantially reduced the paired sample. On SDT tasks, Claude Sonnet 4.6 showed a nominally significant decline on pathogenesis (Δ = −0.038, p = 0.006).

4.3. Tool Usage Patterns

Tool usage rates under the tool-augmented condition varied substantially across models (Figure 4a). GPT-5.2 barely utilized the provided tools (2%, 37/1900 items), whereas Claude Haiku 4.5 (30%) and GPT-5-mini (40%) showed selective usage. Claude Sonnet 4.6 was the most active (87%, average 2.6 calls/item).

API costs scaled with tool usage rate (Figure 4b). GPT-5-mini under the tool-augmented condition incurred costs comparable to GPT-5.2 baseline, while achieving similar TCMBench term accuracy (85.6% vs. 84.3%), suggesting that a lightweight model with tool augmentation can match a larger model’s factual performance at similar cost. In contrast, Claude Sonnet 4.6’s tool-augmented condition was approximately 10× more expensive than the GPT models, driven by higher per-token pricing and frequent tool invocations.

Tool-augmented performance profiles across four task axes revealed distinct model strengths (Figure 5). Claude Sonnet 4.6 exhibited the largest overall coverage, while GPT-5-mini with tool augmentation approached large-model performance on TCMBench tasks.

4.4. Retrieval and Tool-Calling Ablations

To isolate the effect of autonomous tool-calling from the metadata itself, we compared three conditions for GPT-5-mini: baseline, non-agentic retrieval (RAG), and agent (

Table 2. Comparison of GPT-5-mini performance across baseline, non-agentic retrieval (RAG), and agent conditions. GPT-5-mini was selected as it showed the largest agent-condition improvement.

Section	n (R)	Baseline	RAG	Agent	Agent–RAG
TCMBench Term	309	75.4%	77.7%	85.6%	+7.9 pp
TCMBench Other	930	71.0%	72.5%	76.7%	+4.2 pp
SDT Patho.	299	0.474	0.494	0.508	+0.014
SDT Syndr.	299	0.517	0.506	0.478	−0.028

). On TCMBench, a consistent pattern of Agent > RAG > Baseline was observed, with the agent outperforming RAG by +7.9 pp on term-related items and +4.2 pp on other items. The RAG condition provided only modest improvement over baseline (+2.3 pp on term), suggesting that autonomous, selective retrieval is substantially more effective than fixed retrieval. On SDT tasks, differences between conditions were small and inconsistent.

We further analyzed performance conditioned on whether models actually invoked tools for each item (

Table 3. TCMBench term-related accuracy conditioned on tool usage.

Model	Tool Used	n	Not Used	n	Baseline
GPT-5.2	72.7%	11	81.4%	317	84.1%
GPT-5-mini	80.3%	71	89.5%	153	82.1%
Claude Sonnet 4.6	91.2%	284	100.0%	18	89.7%
Claude Haiku 4.5	71.5%	137	77.3%	176	73.5%

“Tool Used” = accuracy on items where the model invoked at least one tool; “Not Used” = accuracy on items where no tools were invoked; “Baseline” = accuracy under the baseline (no-tool) condition on the same items. n = number of items in each category.

). Across all models, accuracy on tool-invoked items was consistently lower than on items without tool use, indicating that models selectively invoke tools on uncertain or difficult items rather than uniformly. This selective behavior complicates causal interpretation, as the observed performance differences reflect both the direct effect of tool-provided information and model-specific tool-calling tendencies.

5. Discussion

This study evaluated a tool-augmented LLM agent system for TAM across four models under baseline, RAG, and agent conditions. The main findings were threefold: tool augmentation showed a nominally significant improvement for a lightweight model (GPT-5-mini) on factual retrieval tasks (McNemar p = 0.034) but did not benefit clinical reasoning tasks; autonomous agent-based retrieval outperformed non-agentic RAG (85.6% vs. 77.7% on TCMBench term); and tool usage patterns varied widely across models (2–87%), with moderate usage associated with the most consistent gains.

Tool augmentation effects were strongly dependent on both model scale and task type. GPT-5-mini showed the only nominally significant improvement on factual retrieval tasks (McNemar p = 0.034), while high-baseline models showed no significant changes. However, tool usage rates varied dramatically across models (2–87%), and the conditional analysis (Table 3) revealed that models selectively invoke tools on uncertain items, complicating causal interpretation. On SDT clinical reasoning tasks, tool augmentation was associated with slight declines, with Claude Sonnet 4.6 showing a nominally significant decrease on pathogenesis (p = 0.006), representing the strongest individual effect observed across all comparisons. This task-type dependency is consistent with the knowledge-practice gap identified by Bedi et al. [28]: the metadata database provides factual information that directly supports recall but contributes less to clinical reasoning requiring synthesis of patient presentations with theoretical frameworks. Future tool designs could incorporate procedural knowledge such as diagnostic reasoning pathways and syndrome differentiation decision trees [21,29].

The three-way comparison (Table 2) demonstrated that autonomous agent-based retrieval consistently outperformed non-agentic RAG on factual tasks (+7.9 pp over RAG on term items), suggesting that selective, query-targeted retrieval is substantially more effective than indiscriminate fixed retrieval [14,15]. Tool usage patterns further revealed distinct model-specific behaviors: GPT-5.2 rarely invoked tools (2%), reflecting high parametric confidence, while Claude Sonnet 4.6’s aggressive usage (87%) required prompt-based control to prevent context window saturation and performance degradation. GPT-5-mini’s moderate usage (40%) was associated with the most consistent gains, aligning with the principle that effective tool use requires selective invocation [16,17].

From a cost perspective, GPT-5-mini with tool augmentation achieved factual accuracy comparable to GPT-5.2’s baseline at similar cost, while Claude Sonnet 4.6’s tool-augmented condition was approximately 10× more expensive, driven by higher per-token pricing and frequent tool invocations (Supplementary Table S1, Figure 4b). For cost-sensitive applications, a lightweight model with tool augmentation may represent a viable alternative to deploying a large model, though this comparison is limited to per-token API pricing.

This study has several limitations. The answer extraction failure rate varied across models, with GPT-5-mini showing a particularly high rate under tool augmentation (26.1%), potentially introducing selection bias. Despite this attrition, GPT-5-mini’s improvement remained nominally significant on the reduced paired sample (McNemar p = 0.034, n ≈ 245 scorable term items out of 331 total), suggesting the effect is robust to the sample reduction. The SDT F1 metric relies on string matching and may underestimate performance for paraphrased responses. As an exploratory analysis, results reaching p < 0.05 were reported as nominally significant even when the Bonferroni-corrected threshold (α = 0.003 for 16 comparisons) was not met. Future work with expanded model coverage, larger benchmark sets, and repeated sampling could strengthen the statistical power for more definitive conclusions. Additionally, the Gemini model family was excluded, and the RAG comparison was limited to GPT-5-mini only. The evaluation code and metadata database are planned for public release upon acceptance.

6. Conclusions

This study provides empirical evidence that tool-augmented LLM agents can improve factual retrieval in the TAM domain, particularly for lightweight models, while autonomous tool invocation outperforms fixed retrieval strategies. Furthermore, we identified that tool-calling frequency is a critical factor influencing agent performance, and that moderate, selective invocation yields the most consistent gains. These findings highlight the need for task-aligned tool design incorporating procedural and clinical knowledge, and model-specific prompt optimization for tool invocation control when developing medical AI agent systems.