Benchmarking Multiple Large Language Models for Automated Clinical Trial Data Extraction in Aging Research

Abstract

Large-language models (LLMs) show promise for automating evidence synthesis, yet head-to-head evaluations remain scarce. We benchmarked five state-of-the-art LLMs—openai/o1-mini, x-ai/grok-2-1212, meta-llama/Llama-3.3-70B-Instruct, google/Gemini-Flash-1.5-8B, and deepseek/DeepSeek-R1-70B-Distill—on extracting protocol details from transcranial direct-current stimulation (tDCS) trials enrolling older adults. A multi-LLM ensemble pipeline ingested ClinicalTrials.gov records, applied a structured JSON schema, and generated comparable outputs from unstructured text. The pipeline retrieved 83 aging-related tDCS trials—roughly double the yield of a conventional keyword search. Across models, agreement was almost perfect for the binary field brain stimulation used (Fleiss κ ≈ 0.92) and substantial for the categorical primary target (κ ≈ 0.71). Numeric parameters such as stimulation intensity and session duration showed excellent consistency when explicitly reported (ICC 0.95–0.96); secondary targets and free-text duration phrases remained challenging (κ ≈ 0.61; ICC ≈ 0.35). An ensemble consensus (majority vote or averaging) resolved most disagreements and delivered near-perfect reliability on core stimulation attributes (κ = 0.94). These results demonstrate that multi-LLM ensembles can markedly expand trial coverage and reach expert-level accuracy on well-defined fields while still requiring human oversight for nuanced or sparsely reported details. The benchmark and open-source workflow set a solid baseline for future advances in prompt engineering, model specialization, and ensemble strategies aimed at fully automated evidence synthesis in neurostimulation research involving aging populations. Overall, the five-model multi-LLM ensemble doubled the number of eligible aging-related tDCS trials retrieved versus keyword searching and achieved near-perfect agreement on core stimulation parameters (κ ≈ 0.94), demonstrating expert-level extraction accuracy.

1. Introduction

Continual advances in medicine rely on evidence-based knowledge [1], which continues to grow at an unprecedented pace. Clinical trials are a cornerstone of modern medical research and inform best practices by providing crucial data on the safety and efficacy of interventions [2]. Although platforms such as ClinicalTrials.gov aggregate hundreds of thousands of trials worldwide [3], systematically extracting and synthesizing their information remains a formidable challenge [4]. Traditional review methods—keyword-based searches and manual data extraction—can take months to complete and often fail to capture critical details found in unstructured protocol text [5,6,7].

Large language models (LLMs) offer a promising avenue for automating this labor-intensive process [8]. Recent models—including openai/o1-mini [9], x-ai/grok-2-1212, meta-llama/llama-3.3-70b-instruct [10], google/gemini-flash-1.5-8b [11], and deepseek/deepseek-r1-distill-llama-70b [12]—exhibit advanced language understanding that surpasses simple keyword matching. Nonetheless, concerns about accuracy, reliability, and “hallucinations” persist [13,14,15,16,17]. As a result, multi-agent systems, in which several specialized LLMs collaborate to tackle different parts of the extraction workflow, have emerged as a compelling approach [18,19,20,21,22]. While single LLMs are commonly benchmarked on general tasks [23,24], their combined performance for domain-specific use cases, such as analyzing aging-related protocols, remains poorly understood and often requires additional strategies like retrieval-augmented generation (RAG).

To shed light on these challenges, we systematically compared the outputs of multiple LLMs for data extraction in clinical trials examining transcranial direct current stimulation (tDCS) in aged populations [25], a domain featuring evolving terminology and varied interventions. In brief, tDCS is a non-invasive brain stimulation technique that can modify cortical excitability and enhance motor skill acquisition in healthy young [26,27,28,29,30] and old adults when delivered to regions of the brain, such as the cerebellum [31], supplemental motor area [32], dorsolateral prefrontal cortex [33], and primary motor cortex. Although a nontrivial minority of studies do not show these positive effects [29,34,35,36,37,38], the balance of the research in young and old adults demonstrates motor skill enhancements of about 10–15% when tDCS is delivered at an intensity of 1–2 mA for 20 min before or during motor task performance. Despite growing interest, the heterogeneity of study protocols including stimulation parameters, targeted brain regions, and outcome measures makes systematic synthesis of this field particularly challenging. For instance, a simple keyword search such as “Aged AND tDCS” often retrieves only a fraction of relevant studies, overlooking important trials due to variations in terminology and inconsistent reporting. Moreover, vital protocol details frequently reside within unstructured text, requiring extensive human curation for accurate extraction [39,40]. Consequently, aging-related tDCS trials serve as an ideal model topic for testing automated data extraction methods, providing valuable insights into the capabilities and limitations of LLM-based approaches within a clinically relevant and complex research area.

In the current study, we automated and standardized significant portions of this data extraction process using a multi-agent LLM pipeline [41]. After downloading and consolidating clinical trial data, each model independently parsed the BriefSummary and DetailedDescription fields before outputting a structured, machine-readable JSON summary. Comparing outputs revealed areas of consensus (e.g., yes/no presence of tDCS interventions) as well as points of ambiguity (e.g., complex parameter descriptions or anatomical targets). This approach not only captured more relevant trials than conventional searches but also demonstrated how multi-agent LLMs can streamline systematic reviews [8,42,43,44,45]. While the findings highlight the benefits of automated pipelines—quicker extraction and reduced manual workload—they also underscore the need for human oversight to reconcile ambiguous information and address the variability of LLM performance. Furthermore, by enabling a rapid iterative refinement of the search criteria and extraction parameters, multi-agent systems can significantly enhance the responsiveness and adaptability of systematic reviews [46] in dynamic research domains. Illustrating both the potential and limitations of this multi-agent paradigm, this work advances aging-related tDCS research and lays the groundwork for applying similar methods to other domains, such as electronic medical records (EMRs), clinical trial guideline synthesis, and broader medical data processing.

This study is among the first to systematically benchmark multiple state-of-the-art LLMs specifically for extracting structured clinical trial data from aging-related research protocols involving neuromodulatory interventions such as tDCS. Unlike previous research, which predominantly assesses single-model performance or focuses on general clinical texts, this investigation uniquely evaluates a multi-agent ensemble approach. By demonstrating the strengths and limitations of individual LLMs and the added value of cross-model consensus methods, this work introduces a novel, rigorous methodology for automated data extraction tailored to the nuanced and heterogeneous domain of aging research.

The primary research question (RQ1) of this study was to examine how consistently five state-of-the-art LLMs—openai/o1-mini, x-ai/grok-2-1212, Llama-3.3-70B-Instruct, Gemini-Flash-1.5-8B, and DeepSeek-R1-70B-Distill—extract key methodological fields from aging-related tDCS trial protocols. We hypothesized substantial, systematic performance differences and expected instruction-tuned, high-parameter models (e.g., Llama-3.3-70B-Instruct, DeepSeek-R1-70B-Distill) to outperform smaller, general-purpose models (e.g., openai/o1-mini) on complex, multi-field extractions involving numeric and multi-arm design details.

The secondary research question (RQ2) investigated whether a simple ensemble consensus method using majority voting for categorical fields and mean or median values for numeric fields could achieve greater accuracy and inter-rater reliability compared to the best-performing individual model. We anticipated that the ensemble method would yield significantly higher agreement with expert-validated reference labels (κ > 0.90 for categorical fields, ICC > 0.95 for numeric fields) than any single model, thus demonstrating the practical value of multi-agent pipelines.

The final research question (RQ3) assessed which attribute classes (binary, categorical, or continuous numeric) pose the greatest challenge for automated extraction. We predicted that binary attributes, such as “brain-stimulation used,” would show near-perfect agreement (κ ≥ 0.90), whereas continuous numeric parameters (e.g., stimulation intensity, session duration) would exhibit notably lower concordance (ICC ≤ 0.60), which would highlight the inherent difficulties models encounter when normalizing heterogeneous quantitative descriptions.

2. Literature Review

LLMs have recently emerged as powerful tools for automating the extraction of structured information from unstructured clinical texts. Traditional manual data extraction for evidence synthesis remains labor-intensive and prone to human error, creating significant workflow bottlenecks [47]. Recent studies highlight the promising capabilities of LLMs in streamlining clinical trial data extraction. For example, Lai et al. (2025) demonstrated that advanced models such as Claude-3.5 and Moonshot-v1 achieve approximately 95% accuracy in extracting data from complementary medicine trials, with accuracy surpassing 97% when supplemented with expert oversight [47]. Similarly, Jensen et al. (2025) showcased the effectiveness of ChatGPT-4o as a second reviewer in systematic reviews of exercise interventions, which achieved 92.4% accuracy with minimal false-positive outputs [48]. Furthermore, Liu et al. (2025) utilized structured prompting methods that achieved 94.8% accuracy while significantly reducing extraction times to approximately 88 s per trial, which underscores the practical advantages of structured prompts in systematic review contexts [49].

Despite these encouraging findings, the literature on LLM-driven clinical trial data extraction remains relatively underdeveloped, often limited to single-model evaluations or narrow clinical contexts. Multi-model evaluations have predominantly targeted clinical narratives and EHRs. These have achieved accuracy rates exceeding 98% with models such as GPT-4 and Claude variants [49]. Stuhlmiller et al. (2025) further confirmed the potential of LLMs in clinical documentation, notably enhancing medication data completeness from EHRs, though their study did not explicitly address clinical trial protocols [50]. Furthermore, Khan et al. (2024) introduced a collaborative two-LLM workflow which demonstrated substantial improvements in accuracy and reliability over single-model approaches [51]. However, systematic benchmarking across multiple LLMs specifically targeting structured clinical trial data extraction remains sparse. Additionally, specialized domains such as aging research and neuromodulatory interventions have received minimal attention in the current literature. Sun et al. (2024) notably highlighted a critical limitation of LLMs—low performance (approximately 39% accuracy) when extracting complex continuous outcome data—which underscores the ongoing challenges inherent in these specialized research areas [52].

Thus, there is a significant gap in comprehensive, multi-LLM benchmarking studies tailored explicitly to nuanced and domain-specific clinical trial contexts. To date, no research has systematically evaluated multiple state-of-the-art LLMs specifically for extracting structured data from clinical trial documents involving aging populations or neuromodulatory techniques, such as brain stimulation. Furthermore, the accuracy of LLMs in handling continuous numeric outcomes and the benefits of cross-model consensus workflows has been virtually unexplored. The present study directly addresses these gaps by systematically benchmarking multiple LLMs for structured data extraction from aging-related tDCS and other neuromodulatory intervention trials.

3. Materials and Methods

We developed a systematic pipeline for retrieving and analyzing aging-related clinical trial data from ClinicalTrials.gov using its v2 Application Programming Interface (API). The pipeline, implemented in Python, interacts with the API through custom functions designed for efficient pagination, comprehensive coverage, and minimal rate limit errors (see GitHub link for source code).

Initial API queries were structured to capture study design, intervention details, enrollment numbers, and outcome measures (Appendix A). We combined search terms such as “Aged” and “tDCS” and applied regex-based filtering (e.g., “tdcs”), which identified a pool of trials potentially relevant to noninvasive brain stimulation in older adults [53,54,55,56].

A verification system checked the unique National Clinical Trial (NCT) identifiers retrieved against the total count reported by the API in each data collection cycle. Any discrepancy automatically triggered additional retrieval attempts, ensuring that the final dataset comprehensively represents the available trials.

All retrieved records were stored in Parquet format to facilitate quick access and seamless transfer between Jupyter notebooks. For broader compatibility and easier manual review, we also maintained copies in CSV and Excel formats. During data cleaning, we standardized placeholders for missing values (e.g., replacing various empty fields with a consistent NULL), converted enrollment counts into numeric fields, and normalized metadata such as FDA regulatory status and healthy-volunteer indicators.

All core data-processing scripts were written in Python 3.12, leveraging several key libraries:

• pandas [1], for data manipulation;
• Requests, for handling API interactions;
• NumPy [2], for numerical operations;
• OpenRouter, for coordinating multi-LLM (large language model) requests.

Version control was maintained for all scripts, configuration files, and environment details (including Python packages and random seeds), thereby ensuring reproducibility of results.

Table 1. Data Source Transparency.

Description	Details
Data Source	ClinicalTrials.gov
API Endpoint URL	https://clinicaltrials.gov/data-api/api
Query Terms-Keywords	“Aged”
Number of Trials Retrieved	10,030 trials
Inclusion Criteria	Trials explicitly mentioning tDCS, targeting older adults (aged ≥ 65), and involving brain stimulation methods
Exclusion Criteria	Trials not explicitly mentioning tDCS, animal studies, duplicates, and unrelated interventions
Final Number of Trials Analyzed	83 trials
Date of Data Extraction	1 January 2025

Shows the Data Source Transparency which includes; Overview of ClinicalTrials.gov API data extraction, including query parameters, inclusion/exclusion criteria, and the final dataset utilized for analysis.

3.1. Natural Language Processing and Analysis

Our analysis framework incorporated five large language models (LLMs), each accessed through the OpenRouter API infrastructure:

• openai/o1-mini;
• x-ai/grok-2-1212;
• meta-llama/llama-3.3-70b-instruct;
• google/gemini-flash-1.5-8b;
• deepseek/deepseek-r1-distill-llama-70b.

Each model was configured with a maximum context window (up to 120,000 tokens) and a temperature of 0.5 to balance creativity and consistency. Although all LLMs received the same prompt, their architectural differences and unique training corpora led to distinctive extraction tendencies—an advantage that enabled a comparative evaluation of their strengths and weaknesses in identifying clinical trial attributes.

Multi-LLM workflow for clinical trial data review. Each trial’s text is processed by all five models, and results are aggregated for cross-validation (Figure 1).

Prompt Architecture

To achieve structured, consistent outputs from these LLMs, we developed a “final” prompt specifying the following:

1. Contextual Trial Information: Key protocol text, including intervention descriptions and outcome measures.

2. Directive: “Analyze whether brain stimulation was used in this trial. If so, provide details”.

Code Box 1 (json):

{

 “brain_stimulation_used”: “Yes” or “No”,

 “stimulation_details”: {

  “primary_type”: “e.g., tDCS, TMS, tACS, DBS, etc.” or null,

  “is_noninvasive”: true or false,

  “primary_target”: “Primary brain region or null”,

  “secondary_targets”: [“List of secondary regions”] or [],

  “stimulation_parameters”: {

   “intensity”: “e.g., 2mA” or null,

   “duration”: “e.g., 20 min” or null

  }

 },

 “confidence_level”: “High”, “Medium”, or “Low”,

 “relevant_quotes”: [“Direct quotes supporting the analysis”]

}

Strict Output Formatting Rules: Each LLM must return only a single JSON object conforming to the schema, without additional explanatory text.

This strict schema enables automated parsing, validation, and comparison of outputs across multiple LLMs. When relevant quotes are provided, they further ground the extracted data in the original protocol text, enhancing transparency and reproducibility. The prompt was iteratively tested across multiple LLM versions to ensure both syntactic and semantic accuracy. Specifically, format checks verified strict JSON syntax, with the exact schema fields and content checks ensuring proper mapping of the fields (e.g., “primary_type”, “is noninvasive”) to trial text.

3.2. Statistical Analysis of Multi-LLM Reliability

We treated the five LLMs as independent “raters” of each trial’s key attributes. Depending on the data type, categorical fields (e.g., presence/absence of brain stimulation) were evaluated using Fleiss’ kappa (κ_F) or Krippendorff’s alpha; continuous fields (e.g., stimulation intensity, duration) were evaluated using the intraclass correlation coefficient (ICC).

For nominal outcomes such as the specific stimulation modality (e.g., tDCS, TMS), we computed Fleiss’ kappa to assess overall agreement among all five models. For numeric parameters, we employed ICC(2,1) (a single measure) or ICC(2,k) (average measures), in line with standard multi-rater reliability approaches in clinical and behavioral research.

We derived consensus (“ensemble”) fields in two ways:

• Majority voting for categorical variables (e.g., “Yes” vs. “No” for brain_stimulation_used).
• The mean or median for numeric fields (e.g., stimulation intensity or session duration). These consensus fields serve as the “best estimates” in downstream analyses, including systematic reviews and meta-analyses.

3.3. Analysis Pipeline

Following data cleaning and initial filtering, each trial’s BriefSummary or DetailedDescription text was processed in parallel by the five LLMs. This multi-model approach leverages the unique strengths of each model while minimizing the impact of omissions or misinterpretations by any single LLM.

A specialized subsystem then conducts the following:

• Parses the returned JSON outputs.
• Validates the JSON structure (e.g., required fields and data types).
• Flags malformed outputs for potential manual review.
• Stores validated results in standardized columns, allowing for direct comparison across models.

Comprehensive logging includes request parameters, response times, and errors to facilitate auditing. Automatic retry logic handles transient network or rate limit failures, enabling stable large-scale operation.

3.4. Reproducibility

Several rigorous measures were employed to ensure the reproducibility of this study. Fixed random seeds (seed: 10031975) were consistently used across all steps involving stochastic processes. Detailed logging captured comprehensive records of all API interactions, including request URLs, response codes, and timestamps. Additionally, all scripts and pipeline configurations were maintained under version control, enabling exact replication of computational environments and procedures. Collectively, these strategies guarantee consistent outcomes across repeated analyses and over time, thereby strengthening the reliability and validity of our multi-LLM framework for clinical trial data extraction.

3.5. Alignment of Research Questions with Analytical Approaches

RQ1: 

How consistently do five state-of-the-art LLMs (openai/o1-mini, x-ai/grok-2-1212, Llama-3.3-70B-Instruct, Gemini-Flash-1.5-8B, DeepSeek-R1-70B-Distill) extract key methodological fields from aging-related tDCS trial protocols?

Analysis Method and Justification: Fleiss’ Kappa (κ), for categorical data, and intraclass correlation coefficient (ICC), for numeric data, were selected.

Justification: Fleiss’ Kappa is ideal for assessing agreement among multiple raters (LLMs) when classifying categorical variables. ICC is appropriate for measuring reliability in extracting numeric data, as it assesses the consistency or conformity of quantitative measurements made by different observers (LLMs).

RQ2: 

Does a simple ensemble consensus (majority vote for categorical fields; mean/median for numeric fields) outperform the best individual model in accuracy and inter-rater reliability?

Analysis Method and Justification: Ensemble consensus metrics (majority vote for categorical and averaging methods for numeric variables) were compared directly to individual LLM performance using κ and ICC.

Justification: Ensemble methods typically outperform individual classifiers due to a reduction in individual biases and errors. Majority voting for categorical fields and averaging numeric fields are robust consensus-building methods, and comparing these to the best-performing individual model helps clarify the practical advantage of multi-model ensemble approaches.

RQ3: 

Which attribute classes (binary, categorical, and continuous numeric) pose the most significant challenge to automated extraction?

Analysis Method and Justification: Performance metrics (Fleiss’ Kappa and ICC) were calculated separately for binary, categorical, and numeric fields, facilitating direct comparison across attribute classes.

Justification: Analyzing attribute-specific extraction difficulties provides targeted insights for future methodological refinements. By differentiating attribute classes, the analysis explicitly identifies where LLM-driven pipelines might require additional human oversight or prompt engineering improvements.

The selected analytical framework systematically evaluates model performance, captures inter-model variability, and clearly distinguishes individual versus ensemble outcomes. This structured approach ensures transparency and reproducibility, while explicitly aligning each research question with an appropriate, well-established statistical measure.

Code Availability

The Python code that generated and processed these data—including API calls, data cleaning routines, and multi-LLM evaluation scripts—is available in the Appendix B and online via GitHub https://github.com/ricyoung/LLM-Pipeline-for-Clinical-Trial-Data-Extraction; accessed on 2 March 2025.

4. Results

The multi-model pipeline analyzed a total of 83 aging-related clinical trials that potentially involved noninvasive brain stimulation, yielding systematically extracted data from five large language models (LLMs). To contextualize the consensus across models,

Table 2. Type of brain stimulation used. # Trials: The number of trials that had non-missing data for that field.

NCTId	# Models Saying Yes	# Models Saying No	Consensus
NCT05511259	3	2	Yes
NCT06501755	4	0	Yes

(Brain Stimulation Used) illustrates the first 10 trials’ results. Here, each row indicates how many models returned a “Yes” or “No” for the presence of brain stimulation, along with the final consensus and the number of discrepant models. Notably, all trials except one (NCT05511259) elicited unanimous agreement, underlining the relative clarity of references to stimulation methods in protocol descriptions. Moving on to

Table 3. Primary type, intensity, and confidence.

NCTId	Model	Primary Type	Intensity	Confidence
NCT06658795	OpenAI_o1_mini	tDCS	None	High
NCT06658795	x_ai_grok_2_1212	tDCS	None	High
NCT06658795	Meta_LLaMA_3.3_70B	tDCS	None	High
NCT06658795	Google_Gemini_Flash_1.5_8b	tDCS	None	High
NCT06658795	DeepSeek_Distill_70b	tDCS	None	High
NCT03814304	OpenAI_o1_mini	tDCS	None	High
NCT03814304	x_ai_grok_2_1212	tDCS	None	High
NCT03814304	Meta_LLaMA_3.3_70B	tDCS	None	High
NCT03814304	Google_Gemini_Flash_1.5_8b	tDCS	None	High
NCT03814304	DeepSeek_Distill_70b	tDCS	2mA	High
NCT02436915	OpenAI_o1_mini	tDCS	None	High
NCT02436915	x_ai_grok_2_1212	tDCS	None	High
NCT02436915	Meta_LLaMA_3.3_70B	tDCS	None	High
NCT02436915	Google_Gemini_Flash_1.5_8b	tDCS	None	High
NCT02436915	DeepSeek_Distill_70b	tDCS	2mA	High

(primary type, intensity, and confidence), we include each LLM’s parsing of the critical attributes—primary type, intensity, and confidence for the first five unique NCT identifiers. In most instances, the models converge on “tDCS,” though some records incorporate variations (e.g., “ctDCS,” “TMS”). The intensity column often remains blank, or partially populated (e.g., “2 mA”) when the text provides a clear numeric detail. Confidence levels here trend strongly toward “High” (Figure 2) but the variability across models becomes more evident as additional trials are analyzed.

Table 2 presents the two sample trials (NCT05511259 and NCT06501755) where the five models did not unanimously vote “Yes” or “No”.

Table 3 presents each model’s extracted Primary Type of brain stimulation, its intensity (e.g., 2 mA), and a confidence rating for the first five unique NCT IDs. It demonstrates how different large language models parse complex clinical trial descriptions.

Turning to overall inter-model concordance,

Table 4. This table compares how consistently five large language models (LLMs) label key fields in clinical trial text. We report field the attribute being extracted (e.g., whether or not brain stimulation is used, the primary stimulation type, etc.). Fleiss’ kappa: a statistical measure of how well the models agree beyond chance. Values typically range from −1.0 to 1.0. Mean % Agreement: The average fraction of model outputs that match the majority label in each trial. # Trials: The number of trials that had non-missing data for that field. Categories: The distinct labels (e.g., ‘yes’, ‘no’, ‘tDCS’) that appeared in the model outputs for that field.

Field	FleissKappa	Mean % Agreement	# Trials	Categories
brain_stimulation_used	0.941	0.993	83	yes, no
primary_type	0.709	0.938	78	ctdcs, dtms, dtms, tdcs, hd-tdcs, missing, non-invasive brain stimulation, tacs, tcs, tdcs, tdcs and tacs, tdcs, tacs, tms, tps, tps (transcranial pulse stimulation), tps (transcranial pulse stimulation, also known as low-intensity extracorporeal shock wave therapy (li-eswt))
is_noninvasive	−0.003	0.978	81	true, false
primary_target	0.53	0.747	76	anodal, anodal transcranial direct current stimulation (atdcs), audio-visual associative memory areas, bifrontal, brain regions associated with cognitive function, brain regions associated with sleep spindles, brain regions involved in active cognitive function, brain regions underneath the neocortex, brain underneath the neocortex, center electrode, central nervous system, cerebellum, cerebral cortex, cognitive control network (ccn), cortical and deep brain structures, cortical areas, corticothalamic and corticospinal projections, dlpfc, dorso-lateral prefrontal cortex, dorso-lateral prefrontal cortices, dorsolateral prefrontal cortex, dorsolateral prefrontal cortex (dlpfc), dorsolateral prefrontal cortex (f3), dorsolateral prefrontal cortex, areas of the memory and language network, dorsolateral prefrontal cortex, memory and language network, f3, frontal brain regions, frontal circuits, fronto-central, fronto-central alpha, fronto-central alpha region, fronto-central region, frontopolar cortex, inferior frontal lobe, inferior frontal lobe and superior parietal lobe, knee oa-related areas, language areas, language areas of the brain, lateral prefrontal cortex (lpfc), lateral prefrontal cortex (lpfc) and default mode network (dmn), left dlpfc, left dorsal lateral prefrontal cortex (dlpfc), left dorsolateral prefrontal cortex, left dorsolateral prefrontal cortex (dlpfc), left m1, left prefrontal cortex, m1, m1 (primary motor cortex), memory-related brain regions, mid-cingulate cortex (mcc), missing, motor cortex, neocortex, not explicitly stated, not specified, null, pain-related brain regions, pf areas, pfc, pre-frontal (pf) areas, pre-frontal (pf) brain areas, pre-frontal areas, precuneus, prefrontal circuits, prefrontal cortex, prefrontal cortex and angular gyrus, prefrontal cortex, angular gyrus, prefrontal cortex, specifically the dlpfc, prefrontal cortical activity, prefrontal regions f3/f4, prefrontal right region, primary motor cortex, primary motor cortex (m1), primary motor cortex (m1) contralateral to the moving leg (cm1), primary motor cortex (m1), posterior parietal cortex (ppc), and cerebellar cortex (cbm), right dlpfc and right ppc, right dorsolateral prefrontal cortex (dlpfc), right dorsolateral prefrontal cortex (dlpfc) and right posterior parietal cortex (ppc), right inferior frontal lobe, right temporoparietal junction, right temporoparietal junction (rtpj), right temporoparietal junction (rtpj) or dorsomedial prefrontal cortex (dmpfc), scalp, subcortical areas, theta frequency, unspecified
parameters_intensity	0.498	0.786	28	1 ma, 1–2 ma, 1–2 ma (tdcs), 1–2 milliampere (ma), 1.5 ma, 1.5 ma, 1 ma, 2 ma, 2 milliampere (ma), 2 ma, 3 ma, 3 ma, anodal stimulation, low-intensity, missing, not specified, null, very low energy, very weak electrical current, very weak electrical current from a 9-volt battery
parameters_duration	0.498	0.765	40	1 h/week, 1 h/week for 10 weeks, 1 min, 1 min, 1 min on, 5 s off, 10 sessions over 2 weeks, 13 min, 13 min, 1 min with 5 s breaks, 2 weeks, 20 min, 20 to 60 min, 20–30 min, 10 days, and 2 weeks, 3 sessions per week for 2–4 weeks, 3 sessions per week for 3 weeks (9 sessions total), 3 sessions per week, 6000 pulses each, for 2–4 weeks, 30 min, 30 min for 5 successive days for temporal cortex stimulation, 30 min per session, 30 min per session, 10 sessions over two weeks, 4 weeks, 4-week (3 times per week) treatment, 4000 pulses each, every 6 months for an average of two to four years; 3 tps sessions (6000 pulses each) per week for 2–4 weeks, 5 consecutive days, 5 consecutive days, 30 min per session, 5 sessions/day over 4 days, 5 sessions/day over 4 days only, 6 sessions over 2 weeks, 8 sessions of anodal tdcs completed twice a week for 4 weeks, 8 sessions, twice a week for 4 weeks, 9 sessions (3 sessions per week for 3 weeks), acute (one-time), applied during training sessions, approximately 20 min, daily over 4 weeks, during slow wave sleep, missing, null, repeated sessions, two weeks, two-week, varied (e.g., 3 times per week for 2–4 weeks)

(High-Level Inter-Model Reliability Summary, Debug Edition) lists Fleiss’ kappa (κ) and the mean percent agreement for six fields: for brain_stimulation_used, the near-perfect κ (~0.94) and ~99% mean agreement underscore how readily the models identify transcranial stimulation; for primary_type, there is a substantial κ (~0.71) and >93% mean agreement, reflecting minor confusion around acronyms (e.g., “tCS,” “TPS”); for is_noninvasive, despite the ~98% raw agreement, κ (~−0.003) hovers near zero due to class imbalance—all but a few interventions were classified as noninvasive; for primary_target, there is a moderate κ (~0.53) with ~75% agreement, driven by the variety of anatomical terms and the partial mention of target sites; for parameters_intensity and parameters_duration, both hovered around κ ≈ 0.50, with 75–78% agreement, reflecting variability in numeric references (“2 mA,” “very weak current,” “20 to 60 min,” etc.).

In this analysis, we evaluated categorical and numeric fields extracted by five LLMs. We added two additional fields beyond the original set confidence_level (categorical: High, Medium, or Low) and frequency (numeric, e.g., 140 from “140 Hz” or 2 from “2 Hz”).

Table 5. Inter-model reliability and agreement statistics.

(a)
Field	FleissKappa	95% CI	Interpretation
brain_stimulation_used	0.904	0.68–1.0	Almost Perfect
primary_type	0.709	0.6–0.82	Substantial
is_noninvasive	0.59	0.4–0.71	Moderate
(b)
Parameter	ICC(2,1)	95% CI	Interpretation
intensity	0.95	0.9–0.98	Excellent
duration	0.964	0.94–0.98	Excellent
frequency	1	1.0–1.0	Excellent

(a) Fleiss’ kappa for categorical fields (bootstrap CI): each categorical field was converted into a contingency matrix (N×k), and Fleiss’ κ was computed using a naive bootstrap (1000 resamples) to derive approximate 95% confidence intervals. (b) ICC for numeric fields (Pingouin, ICC2): Numeric fields were parsed as floats (e.g., “140 Hz” → 140.0) and processed via pingouin’s intraclass_corr to obtain the ICC(2) (two-way random, single measure). We report the ICC estimate, 95% CI, and a qualitative interpretation (Excellent, Good, Moderate, or Poor).

(Inter-model reliability and agreement statistics) provides a more detailed breakdown, splitting the reliability metrics into (A) Fleiss’ kappa for categorical fields, with bootstrap confidence intervals, and (B) ICC (intraclass correlation coefficient) for numeric data. “brain_stimulation_used” achieves an Almost Perfect κ (0.90); “primary_type” (κ ≈ 0.71) remains Substantial; and “is_noninvasive” (κ ≈ 0.59) stands at Moderate.

For numeric parameters (intensity, duration, frequency), ICC values range from 0.95 to 1.0 (Excellent), indicating strong consistency when the text clearly presents numerical values. At the same time, an ICC of 1.0 may reflect limited variability in how frequency is reported—many trials either did not provide frequency or gave a simple measure (e.g., “2 Hz”).

Table 6. Final consensus table—first.

NCT ID	Brain Stimulation	Stimulation Type	Intensity (mA)	Duration (Minutes)	Primary Target	Confidence	Unanimous?	Comments
NCT06658795	Yes	tDCS			lateral prefrontal cortex (LPFC)	High	No	Intensity (mA) note: no numeric parse; duration (minutes) note: no numeric parse
NCT05511259	Yes	TMS			None	High	No	Intensity (mA) note: no numeric parse; duration (minutes) note: no numeric parse
NCT04154397	Yes	ctDCS			cerebellum	High	No	Intensity (mA) note: no numeric parse; duration (minutes) note: no numeric parse
NCT03814304	Yes	tDCS	2.00	20.00	left dorsolateral prefrontal cortex (dlPFC)	High	No
NCT02436915	Yes	tDCS	2.00	20.00	left prefrontal cortex	High	No

(Final “Consensus” Table) illustrates how each trial’s fields look after majority voting for categorical variables and mean (or most common) values for numeric variables. Trials like NCT03814304 show a clear consensus (“tDCS,” 2.0 mA, 20.0 min), whereas others display blank or partial columns when protocols lack explicit numeric details.

Table 7. Per-field agreement vs. majority.

Model Suffix	Brain Stim.	Primary Type	Is Noninvasive	Intensity	Duration	Primary Target	Avg. Agreement
GPT-o1-mini	98.8	90.2	95.2	77.8	68.8	87.3	86.3
x.ai Grok	100	96.3	100	77.8	90.6	75.9	90.1
Meta LLaMA 3.3 70B	98.8	92.7	98.8	77.8	90.6	86.1	90.8
Google Gemini Flash 1.5B	98.8	96.3	97.6	77.8	78.1	83.5	88.7
DeepSeek Distill 70B	100	98.8	97.6	94.4	90.6	50.6	88.7

(Per-field agreement vs. majority) presents how often each LLM’s output matches the consensus label for each field. Binary fields like “Brain Stim.” and “Is Noninvasive” achieve ≥90% agreement in nearly all cases, with some models (x.ai Grok and DeepSeek Distill 70B) reaching 100%. Numeric fields such as “Intensity” and “Duration” show lower alignment (68–94%), reflecting inconsistencies in how dosage or session length is described. “Primary Target” also remains moderately challenging: while most models exceed 75%, DeepSeek Distill 70B dips near 50%, possibly indicating heightened sensitivity to ambiguous text or multiple target mentions within a single protocol. Overall, these results confirm that the multi-agent LLM pipeline reliably classifies fundamental trial features, though textual ambiguity still introduces moderate variation, especially in multi-arm or highly detailed protocols.

Figure 3 provides detailed pairwise comparisons among models for the binary field ‘Brain Stimulation Used’. The high Cohen’s kappa values (≥0.90) indicate excellent inter-model reliability, further validating the robustness of the LLM-based extraction process. Confusion matrices reveal minor discrepancies, which predominantly occurred in only a few isolated cases.

This heatmap displays a symmetrical 5 × 5 matrix representing the pairwise agreement (e.g., Cohen’s kappa) among the five large language models (LLMs). Each cell indicates how similarly two models classified or extracted data across all trials. Darker shades (closer to 1.0) reflect near-perfect consistency, while lighter shades reveal greater discrepancies. The diagonal entries, set to 1.0, represent each model’s agreement with itself.

5. Discussion

The primary findings of this study demonstrate that individual large language models (LLMs) exhibit distinct strengths and weaknesses when extracting key parameters from clinical trial protocols involving transcranial direct current stimulation (tDCS) in aging research [57,58,59,60,61]. Consistent with previous research [11,12,62], all tested models reliably identified fundamental binary attributes, such as the presence or absence of brain stimulation and whether interventions were invasive or noninvasive, achieving near-perfect to substantial agreement. However, considerable variability was observed in extracting more nuanced or numeric protocol features, such as secondary anatomical targets or session durations, reflecting inherent differences in model architectures and training strategies [63,64].

The observed differences in performance across large language models (LLMs) likely stem from their underlying architectures, parameter sizes, and training corpora. Larger, instruction-tuned models such as Meta Llama 3.3–70B and DeepSeek-Distill 70B consistently performed better in extracting complex numeric details and nuanced descriptions due to their extensive training on large, domain-specific datasets, which likely provide them with richer contextual embeddings. In contrast, smaller, general-purpose models (e.g., openai/o1-mini, Google Gemini-Flash-1.5-8B) struggled with ambiguous language or imprecise numeric references, likely because they rely heavily on general linguistic patterns without sufficient domain-specific exposure. Additionally, the moderate agreement seen in identifying anatomical targets appears rooted in protocol ambiguity, where indirect references (such as “regions associated with cognitive function”) create inherent interpretative challenges. The strength of ensemble methods observed here can be attributed to their ability to compensate for individual model biases and inconsistencies, highlighting their advantage in reducing errors through consensus. Finally, persistent variability and occasional inaccuracies among all models underscore inherent limitations in current LLM technology, emphasizing the critical role of continued human oversight, particularly for ensuring reliability in complex, clinically nuanced data extraction.

As demonstrated in Table 6, majority voting and consensus strategies effectively distilled individual LLM outputs into a coherent record, facilitating rapid identification and appraisal of relevant trial data. Furthermore, the consistent patterns observed in Table 7, where inter-model agreement varied by field complexity, underscore the importance of careful curation and targeted prompt engineering to improve extraction accuracy, specially Figure 4, for complex protocol elements. The κ and ICC scores reported here translate directly into practical guidance for implementing automated extraction pipelines in systematic reviews. High agreement metrics affirm the capability of multi-LLM ensembles to substantially streamline the extraction of clearly defined protocol details. However, lower reliability observed for ambiguous or sparsely documented attributes underscores the continued necessity of structured human oversight. Thus, researchers should leverage these findings to strategically integrate human validation points, especially for nuanced protocol elements. This would ensure the robust applicability of LLM-based systems in real-world settings.

The significant variation among models on complex attributes highlights a critical consideration for future multi-agent system (MAS) design [65,66,67] by selecting complementary models to leverage their individual strengths effectively. For example, models like DeepSeek-Distill [68] showed high accuracy in extracting numeric parameters such as intensity and duration, whereas models such as Meta Llama excelled in identifying multi-arm trial designs or detailed anatomical descriptions [69,70]. These insights underscore the value of a combined approach in future MAS development, tailored to specific data extraction tasks to optimize accuracy and consistency.

Leveraging these model capabilities can significantly advance aging research by providing clearer insights into current clinical trials, enabling medical professionals to better understand aging processes and variability across different populations [71,72,73]. Furthermore, optimized LLM-based systems could greatly improve patient education [74,75], providing access not only to foundational knowledge but also to current and promising clinical trials. This approach could facilitate better communication among patients, researchers, and medical providers, thereby enhancing the overall effectiveness of aging interventions [76].

A practical implication for aging research is that multi-agent LLM pipelines can significantly enhance the current understanding of aging processes by systematically identifying the current state of clinical trials and highlighting variations across different populations [77]. Clinically tuned LLM systems could greatly improve patient education, as they can incorporate up-to-date clinical trial data and provide accessible, detailed information to patients, researchers, and medical providers alike.

This comparative analysis not only establishes benchmarks for LLM performance in clinical trial extraction, but also emphasizes the ongoing necessity of human oversight to reconcile ambiguous outputs and address limitations inherent in current LLM technology [78], including susceptibility to misinterpretation and hallucination [14,79,80]. While RAG strategies offer promising enhancements for accuracy, implementing RAG with extensive clinical trial datasets is not always practical or feasible. Consequently, future research should focus on optimizing foundational LLMs through specialized prompt engineering and domain-specific ontologies [81]. A deeper understanding of foundational model performance will guide effective RAG design choices, ensuring that multi-agent systems developed are precise, practical, and well suited to advanced clinical applications and precision medicine.

Recognizing these ethical and practical imperatives, we implemented an explicit human-in-the-loop framework within our multi-LLM ensemble pipeline. Although our approach is not a true multi-agent system where agents dynamically collaborate, but rather an ensemble of independently operated LLMs, structured human oversight remains integral. This framework included three essential components: (1) periodic manual audits, conducted regularly on randomly selected subsets of extracted data to identify systematic errors or misinterpretations by individual models; (2) defined human-in-the-loop checkpoints, strategically positioned after initial data extraction and prior to consensus formation, ensuring reviewers could systematically review and resolve ambiguous or conflicting outputs; and (3) automated triggers for human intervention, activated by predefined thresholds such as low-confidence scores, significant discrepancies among model outputs, or parsing anomalies. Our pipeline was intentionally structured into three distinct processing stages (trial retrieval, initial filtering, and structured data extraction) with automated halts designed explicitly to facilitate careful human validation before advancing. This structured integration of human oversight addresses critical ethical considerations, enhances data reliability, and bolsters the practical validity of the high κ/ICC scores reported, underscoring the real-world applicability of our results in systematic reviews and clinical decision-making.

6. Conclusions

This study provides a systematic comparative analysis of several large language models (LLMs) for clinical trial data extraction in aging research, revealing distinct performance profiles that can inform future multi-agent system (MAS) development. Although all evaluated models performed exceptionally well on straightforward attributes, considerable variability was noted for complex or numeric parameters. Recognizing these differences enables researchers to strategically select complementary models for MAS implementation, ensuring that each model’s strengths are optimally leveraged.

The clinical value of such optimized MAS frameworks is substantial, as they can rapidly synthesize extensive clinical trial data, ultimately enhancing patient care and provider education. Future work should extend this comparative framework by integrating external data sources (e.g., PubMed, NIH Reporter) to enhance context and validation and by developing targeted MAS pipelines where models specialize in different extraction subtasks. Such refined systems could significantly expedite systematic reviews and meta-analyses in clinical aging research, benefiting researchers through improved efficiency and precision. However, careful attention must be given to ethical considerations, transparency, and rigorous human oversight to ensure reliability, particularly as these tools become more integrated into clinical decision-making processes. As LLM capabilities advance, structured guidelines and rigorous validation standards will become essential to maintain data quality, safeguard patient privacy, and promote the responsible adoption of AI-driven methodologies in medical research [82].

Future Research Directions

Future research should focus on designing true multi-agent agentic systems, where individual LLM agents dynamically collaborate and refine their extraction tasks in real-time, significantly enhancing extraction accuracy and efficiency. Integrating additional data sources, such as PubMed publications and other scholarly databases, can provide richer contextual insights, especially when individual clinical trials yield multiple related publications. Moreover, evaluating more advanced and larger models, including GPT-4 variants and Claude 3.7, could offer further improvements in accuracy and reliability, particularly for handling complex, nuanced, or ambiguous clinical data. Such advancements would move the field closer to fully automated, precise, and contextually aware clinical data extraction workflows.